742 lines
120 KiB
XML
742 lines
120 KiB
XML
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.6">Jekyll</generator><link href="http://0.0.0.0:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://0.0.0.0:4000/" rel="alternate" type="text/html" /><updated>2019-11-30T11:30:39-05:00</updated><id>http://0.0.0.0:4000/feed.xml</id><title type="html">Evan Pratten</title><subtitle>Computer wizard, student, <a href="https://frc5024.github.io">@frc5024</a> programming team lead, and radio enthusiast.</subtitle><entry><title type="html">Programming a live robot</title><link href="http://0.0.0.0:4000/blog/2019/11/20/realtime-robot-code" rel="alternate" type="text/html" title="Programming a live robot" /><published>2019-11-20T05:04:00-05:00</published><updated>2019-11-20T05:04:00-05:00</updated><id>http://0.0.0.0:4000/blog/2019/11/20/Realtime-robot-code</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/11/20/realtime-robot-code"><blockquote>
|
||
<p><em>“So.. what if we could skip asking for driver inputs, and just have the robot operators control the bot through a commandline interface?”</em></p>
|
||
</blockquote>
|
||
|
||
<p>This is exactly the kind of question I randomly ask while sitting in the middle of class, staring at my laptop. So, here is a post about my real-time programming adventure!</p>
|
||
|
||
<h2 id="geting-started">Geting started</h2>
|
||
|
||
<p>To get started, I needed a few things. Firstly, I have a laptop running <a href="/about/#my-gear">a Linux distribution</a>. This allows me to use <a href="https://en.wikipedia.org/wiki/Secure_Shell">SSH</a> and <a href="https://en.wikipedia.org/wiki/Secure_copy">SCP</a>. There are Windows versions of both of these programs, but I find the “linux experience” easier to use. Secondly, I have grabbed one of <a href="https://www.thebluealliance.com/team/5024">5024</a>’s <a href="https://cs.5024.ca/webdocs/docs/robots">robots</a> to be subjected to my experiment. The components I care about are:</p>
|
||
|
||
<ul>
|
||
<li>A RoboRIO running 2019v12 firmware</li>
|
||
<li>2 <a href="https://www.ctr-electronics.com/talon-srx.html">TalonSRX</a> motor controllers</li>
|
||
<li>An FRC router</li>
|
||
</ul>
|
||
|
||
<p>Most importantly, the RoboRIO has <a href="https://robotpy.readthedocs.io/en/stable/install/robot.html#install-robotpy">RobotPy</a> and the <a href="https://robotpy.readthedocs.io/en/stable/install/ctre.html">CTRE Libraries</a> installed.</p>
|
||
|
||
<h3 id="ssh-connection">SSH connection</h3>
|
||
|
||
<p>To get some code running on the robot, we must first connect to it via SSH. Depending on your connection to the RoboRIO, this step may be different. Generally, the following command will work just fine to connect (assuming your computer has an <a href="https://en.wikipedia.org/wiki/Multicast_DNS">mDNS</a> service):</p>
|
||
|
||
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh admin@roborio-&lt;team&gt;-frc.local
|
||
</code></pre></div></div>
|
||
|
||
<p>If you have issues, try one of the following addresses instead:</p>
|
||
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>roborio-&lt;team&gt;-FRC
|
||
roborio-&lt;team&gt;-FRC.lan
|
||
roborio-&lt;team&gt;-FRC.frc-field.local
|
||
10.TE.AM.2
|
||
172.22.11.2 # Only works on a USB connection
|
||
</code></pre></div></div>
|
||
|
||
<p>If you are asked for a password, and have not set one, press <kbd>Enter</kbd> 3 times (Don’t ask why.. this just works).</p>
|
||
|
||
<h2 id="repl-based-control">REPL-based control</h2>
|
||
|
||
<p>If you have seen my work before, you’ll know that I use Python for basically everything. This project is no exception. Conveniently, the RoboRIO is a linux-based device, and can run a Python3 <a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>. This allows real-time robot programming using a REPL via SSH.</p>
|
||
|
||
<p>WPILib requires a robot class to act as a “callback” for robot actions. My idea was to build a special robot class with static methods to allow me to start it, then use the REPL to interact with some control methods (like <code class="highlighter-rouge">setSpeed</code> and <code class="highlighter-rouge">stop</code>).</p>
|
||
|
||
<p>After connecting to the robot via SSH, a Python REPL can be started by running <code class="highlighter-rouge">python3</code>. If there is already robot code running, it will be automatically killed in the next step.</p>
|
||
|
||
<p>With Python running, we will need 2 libraries imported. <code class="highlighter-rouge">wpilib</code> and <code class="highlighter-rouge">ctre</code>. When importing <code class="highlighter-rouge">wpilib</code> a message may appear to notify you that the old robot code has been stopped.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">wpilib</span>
|
||
<span class="n">Killing</span> <span class="n">previously</span> <span class="n">running</span> <span class="n">FRC</span> <span class="n">program</span><span class="o">...</span>
|
||
<span class="n">FRC</span> <span class="n">pid</span> <span class="mi">1445</span> <span class="n">did</span> <span class="ow">not</span> <span class="n">die</span> <span class="n">within</span> <span class="mi">0</span><span class="n">ms</span><span class="o">.</span> <span class="n">Force</span> <span class="n">killing</span> <span class="k">with</span> <span class="n">kill</span> <span class="o">-</span><span class="mi">9</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">ctre</span>
|
||
</code></pre></div></div>
|
||
<p>Keep in mind, this is a REPL. Lines that start with <code class="highlighter-rouge">&gt;&gt;&gt;</code> or <code class="highlighter-rouge">...</code> are <em>user input</em>. Everything else is produced by code.</p>
|
||
|
||
<p>Next, we need to write a little code to get the robot operational. To save time, I wrote this “library” to do most of the work for me. Just save this as <code class="highlighter-rouge">rtrbt.py</code> somewhere, then use SCP to copy it to <code class="highlighter-rouge">/home/lvuser/rtrbt.py</code>.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># RealTime FRC Robot control helper
|
||
# By: Evan Pratten &lt;ewpratten&gt;
|
||
</span>
|
||
<span class="c1"># Import normal robot stuff
|
||
</span><span class="kn">import</span> <span class="nn">wpilib</span>
|
||
<span class="kn">import</span> <span class="nn">ctre</span>
|
||
|
||
<span class="c1"># Handle WPI trickery
|
||
</span><span class="k">try</span><span class="p">:</span>
|
||
<span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
|
||
<span class="k">except</span> <span class="nb">ImportError</span><span class="p">:</span>
|
||
<span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">patch</span>
|
||
<span class="kn">import</span> <span class="nn">sys</span>
|
||
<span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span>
|
||
|
||
|
||
<span class="c1">## Internal methods ##
|
||
</span><span class="n">_controllers</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="n">_thread</span><span class="p">:</span> <span class="n">Thread</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">_RTRobot</span><span class="p">(</span><span class="n">wpilib</span><span class="o">.</span><span class="n">TimedRobot</span><span class="p">):</span>
|
||
|
||
<span class="k">def</span> <span class="nf">robotInit</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
|
||
<span class="c1"># Create motors
|
||
</span> <span class="n">_controllers</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ctre</span><span class="o">.</span><span class="n">WPI_TalonSRX</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
|
||
<span class="n">_controllers</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ctre</span><span class="o">.</span><span class="n">WPI_TalonSRX</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span>
|
||
|
||
<span class="c1"># Set safe modes
|
||
</span> <span class="n">_controllers</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">setSafetyEnabled</span><span class="p">(</span><span class="bp">False</span><span class="p">)</span>
|
||
<span class="n">_controllers</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">setSafetyEnabled</span><span class="p">(</span><span class="bp">False</span><span class="p">)</span>
|
||
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">_start</span><span class="p">():</span>
|
||
<span class="c1"># Handle fake args
|
||
</span> <span class="n">args</span> <span class="o">=</span> <span class="p">[</span><span class="s">"run"</span><span class="p">,</span> <span class="s">"run"</span><span class="p">]</span>
|
||
<span class="k">with</span> <span class="n">patch</span><span class="o">.</span><span class="nb">object</span><span class="p">(</span><span class="n">sys</span><span class="p">,</span> <span class="s">"argv"</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
|
||
<span class="k">print</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
|
||
<span class="n">wpilib</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">_RTRobot</span><span class="p">)</span>
|
||
|
||
<span class="c1">## Utils ##
|
||
</span>
|
||
|
||
<span class="k">def</span> <span class="nf">startRobot</span><span class="p">():</span>
|
||
<span class="s">""" Start the robot code """</span>
|
||
<span class="k">global</span> <span class="n">_thread</span>
|
||
<span class="n">_thread</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">_start</span><span class="p">)</span>
|
||
<span class="n">_thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">setMotor</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">speed</span><span class="p">):</span>
|
||
<span class="s">""" Set a motor speed """</span>
|
||
<span class="n">_controllers</span><span class="p">[</span><span class="nb">id</span><span class="p">]</span><span class="o">.</span><span class="nb">set</span><span class="p">(</span><span class="n">speed</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">arcadeDrive</span><span class="p">(</span><span class="n">speed</span><span class="p">,</span> <span class="n">rotation</span><span class="p">):</span>
|
||
<span class="s">""" Control the robot with arcade inputs """</span>
|
||
|
||
<span class="n">l</span> <span class="o">=</span> <span class="n">speed</span> <span class="o">+</span> <span class="n">rotation</span>
|
||
<span class="n">r</span> <span class="o">=</span> <span class="n">speed</span> <span class="o">-</span> <span class="n">rotation</span>
|
||
|
||
<span class="n">setMotor</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">l</span><span class="p">)</span>
|
||
<span class="n">setMotor</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">r</span><span class="p">)</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>The idea is to create a simple robot program with global hooks into the motor controllers. Python’s mocking tools are used to fake commandline arguments to trick robotpy into thinking this script is being run via the RIO’s robotCommand.</p>
|
||
|
||
<p>Once this script has been placed on the robot, SSH back in as <code class="highlighter-rouge">lvuser</code> (not <code class="highlighter-rouge">admin</code>), and run <code class="highlighter-rouge">python3</code>. If using <code class="highlighter-rouge">rtrbt.py</code>, the imports mentioned above are handled for you. To start the robot, just run the following:</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">rtrbt</span> <span class="kn">import</span> <span class="o">*</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">startRobot</span><span class="p">()</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>WPILib will dump some logs into the terminal (and probably some spam) from it’s own thread. Don’t worry if you can’t see the REPL prompt. It’s probably just hidden due to the use of multiple threads in the same shell. Pressing <kbd>Enter</kbd> should show it again.</p>
|
||
|
||
<p>I added 2 functions for controlling motors. The first, <code class="highlighter-rouge">setMotor</code>, will set either the left (0), or right (1) motor to the specified speed. <code class="highlighter-rouge">arcadeDrive</code> will allow you to specify a speed and rotational force for the robot’s drivetrain.</p>
|
||
|
||
<p>To kill the code and exit, press <kbd>CTRL</kbd> + <kbd>D</kbd> then <kbd>CTRL</kbd> + <kbd>C</kbd>.</p>
|
||
|
||
<p>Here is an example where I start the bot, then tell it to drive forward, then kill the left motor:</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Python</span> <span class="mf">3.6.8</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Oct</span> <span class="mi">7</span> <span class="mi">2019</span><span class="p">,</span> <span class="mi">12</span><span class="p">:</span><span class="mi">59</span><span class="p">:</span><span class="mi">55</span><span class="p">)</span>
|
||
<span class="p">[</span><span class="n">GCC</span> <span class="mf">8.3.0</span><span class="p">]</span> <span class="n">on</span> <span class="n">linux</span>
|
||
<span class="n">Type</span> <span class="s">"help"</span><span class="p">,</span> <span class="s">"copyright"</span><span class="p">,</span> <span class="s">"credits"</span> <span class="ow">or</span> <span class="s">"license"</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span><span class="o">.</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">rtrbt</span> <span class="kn">import</span> <span class="o">*</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">startRobot</span><span class="p">()</span>
|
||
<span class="p">[</span><span class="s">'run'</span><span class="p">,</span> <span class="s">'run'</span><span class="p">]</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">472</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">wpilib</span> <span class="p">:</span> <span class="n">WPILib</span> <span class="n">version</span> <span class="mf">2019.2.3</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">473</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">wpilib</span> <span class="p">:</span> <span class="n">HAL</span> <span class="n">base</span> <span class="n">version</span> <span class="mf">2019.2.3</span><span class="p">;</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">473</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">wpilib</span> <span class="p">:</span> <span class="n">robotpy</span><span class="o">-</span><span class="n">ctre</span> <span class="n">version</span> <span class="mf">2019.3.2</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">473</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">wpilib</span> <span class="p">:</span> <span class="n">robotpy</span><span class="o">-</span><span class="n">cscore</span> <span class="n">version</span> <span class="mf">2019.1.0</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">473</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">faulthandler</span> <span class="p">:</span> <span class="n">registered</span> <span class="n">SIGUSR2</span> <span class="k">for</span> <span class="n">PID</span> <span class="mi">5447</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">474</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">nt</span> <span class="p">:</span> <span class="n">NetworkTables</span> <span class="n">initialized</span> <span class="ow">in</span> <span class="n">server</span> <span class="n">mode</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">497</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">robot</span> <span class="p">:</span> <span class="n">Default</span> <span class="n">IterativeRobot</span><span class="o">.</span><span class="n">disabledInit</span><span class="p">()</span> <span class="n">method</span><span class="o">...</span> <span class="n">Override</span> <span class="n">me</span><span class="err">!</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">498</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">robot</span> <span class="p">:</span> <span class="n">Default</span> <span class="n">IterativeRobot</span><span class="o">.</span><span class="n">disabledPeriodic</span><span class="p">()</span> <span class="n">method</span><span class="o">...</span> <span class="n">Override</span> <span class="n">me</span><span class="err">!</span>
|
||
<span class="mi">17</span><span class="p">:</span><span class="mi">53</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">498</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">robot</span> <span class="p">:</span> <span class="n">Default</span> <span class="n">IterativeRobot</span><span class="o">.</span><span class="n">robotPeriodic</span><span class="p">()</span> <span class="n">method</span><span class="o">...</span> <span class="n">Override</span> <span class="n">me</span><span class="err">!</span>
|
||
<span class="o">&gt;&gt;&gt;</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">arcadeDrive</span><span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">)</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">setMotor</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">)</span>
|
||
<span class="o">&gt;&gt;&gt;</span>
|
||
<span class="o">^</span><span class="n">C</span>
|
||
<span class="nb">Exception</span> <span class="n">ignored</span> <span class="ow">in</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">module</span> <span class="s">'threading'</span> <span class="k">from</span> <span class="s">'/usr/lib/python3.6/threading.py'</span><span class="o">&gt;</span>
|
||
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
|
||
<span class="n">File</span> <span class="s">"/usr/lib/python3.6/threading.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1294</span><span class="p">,</span> <span class="ow">in</span> <span class="n">_shutdown</span>
|
||
<span class="n">t</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
<span class="n">File</span> <span class="s">"/usr/lib/python3.6/threading.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1056</span><span class="p">,</span> <span class="ow">in</span> <span class="n">join</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_tstate_lock</span><span class="p">()</span>
|
||
<span class="n">File</span> <span class="s">"/usr/lib/python3.6/threading.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1072</span><span class="p">,</span> <span class="ow">in</span> <span class="n">_wait_for_tstate_lock</span>
|
||
<span class="k">elif</span> <span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">block</span><span class="p">,</span> <span class="n">timeout</span><span class="p">):</span>
|
||
<span class="nb">KeyboardInterrupt</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>The message at the end occurs when killing the code.</p>
|
||
|
||
<h2 id="conclusion">Conclusion</h2>
|
||
|
||
<p>I have no idea why any of this would be useful, or if it is even field legal.. It’s just a fun project for a monday morning.</p></content><author><name></name></author><summary type="html">“So.. what if we could skip asking for driver inputs, and just have the robot operators control the bot through a commandline interface?”</summary></entry><entry><title type="html">Using an RNN to generate Bill Wurtz notes</title><link href="http://0.0.0.0:4000/blog/2019/10/05/billwurtz" rel="alternate" type="text/html" title="Using an RNN to generate Bill Wurtz notes" /><published>2019-10-05T14:54:00-04:00</published><updated>2019-10-05T14:54:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/10/05/BillWurtz</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/10/05/billwurtz"><p><a href="https://billwurtz.com/">Bill Wurtz</a> is an American musician who became <a href="https://socialblade.com/youtube/user/billwurtz/realtime">reasonably famous</a> through short musical videos posted to Vine and YouTube. I was searching through his website the other day, and stumbled upon a page labeled <a href="https://billwurtz.com/notebook.html"><em>notebook</em></a>, and thought I should check it out.</p>
|
||
|
||
<p>Bill’s notebook is a large (about 580 posts) collection of random thoughts, ideas, and sometimes just collections of words. A prime source of entertainment, and neural network inputs..</p>
|
||
|
||
<blockquote>
|
||
<p><em>“If you are looking to burn something, fire may be just the ticket”</em> - Bill Wurtz</p>
|
||
</blockquote>
|
||
|
||
<h2 id="choosing-the-right-tool-for-the-job">Choosing the right tool for the job</h2>
|
||
<p>If you haven’t noticed yet, Im building a neural net to generate notes based on his writing style and content. Anyone who has read <a href="/blog/2018/06/27/becomeranter">my first post</a> will know that I have already done a similar project in the past. This means <em>time to reuse come code</em>!</p>
|
||
|
||
<p>For this project, I decided to use an amazing library by @minimaxir called <a href="https://github.com/minimaxir/textgenrnn">textgenrnn</a>. This Python library will handle all of the heavy (and light) work of training an RNN on a text dataset, then generating new text.</p>
|
||
|
||
<h2 id="building-a-dataset">Building a dataset</h2>
|
||
<p>This project was a joke, so I didn’t bother with properly grabbing each post, categorizing them, and parsing them. Instead, I build a little script to pull every HTML file from Bill’s website, and regex out the body. This ended up leaving some artifacts in the output, but I don’t really mind.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">re</span>
|
||
<span class="kn">import</span> <span class="nn">requests</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">loadAllUrls</span><span class="p">():</span>
|
||
<span class="n">page</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"https://billwurtz.com/notebook.html"</span><span class="p">)</span><span class="o">.</span><span class="n">text</span>
|
||
|
||
<span class="n">links</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r"HREF=\"(.*)\"style"</span><span class="p">,</span> <span class="n">page</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">links</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">dumpEach</span><span class="p">(</span><span class="n">urls</span><span class="p">):</span>
|
||
<span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">:</span>
|
||
<span class="n">page</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">f</span><span class="s">"https://billwurtz.com/{url}"</span><span class="p">)</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span>
|
||
<span class="s">"&lt;/br&gt;"</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">"&lt;br&gt;"</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="s">" "</span><span class="p">)</span>
|
||
|
||
<span class="n">data</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r"&lt;/head&gt;(.*)"</span><span class="p">,</span> <span class="n">page</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">MULTILINE</span><span class="p">)</span>
|
||
|
||
<span class="c1"># ensure data
|
||
</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
|
||
<span class="k">continue</span>
|
||
|
||
<span class="k">print</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
|
||
|
||
|
||
<span class="n">urls</span> <span class="o">=</span> <span class="n">loadAllUrls</span><span class="p">()</span>
|
||
<span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">"Loaded {len(urls)} pages"</span><span class="p">)</span>
|
||
<span class="n">dumpEach</span><span class="p">(</span><span class="n">urls</span><span class="p">)</span>
|
||
|
||
</code></pre></div></div>
|
||
|
||
<p>This script will print each of Bill’s notes to the console (on it’s own line). I used a simple redirect to write this to a file.</p>
|
||
|
||
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 scrape.py <span class="o">&gt;</span> posts.txt
|
||
</code></pre></div></div>
|
||
|
||
<h2 id="training">Training</h2>
|
||
<p>To train the RNN, I just used some of textgenrnn’s example code to read the posts file, and build an <a href="https://en.wikipedia.org/wiki/Hierarchical_Data_Format">HDF5</a> file to store the RNN’s neurons.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">textgenrnn</span> <span class="kn">import</span> <span class="n">textgenrnn</span>
|
||
|
||
<span class="n">generator</span> <span class="o">=</span> <span class="n">textgenrnn</span><span class="p">()</span>
|
||
<span class="n">generator</span><span class="o">.</span><span class="n">train_from_file</span><span class="p">(</span><span class="s">"/path/to/posts.txt"</span><span class="p">,</span> <span class="n">num_epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>This takes quite a while to run, so I offloaded it to a <a href="https://www.digitalocean.com/products/droplets/">Droplet</a>, and left it running overnight.</p>
|
||
|
||
<h2 id="the-results">The results</h2>
|
||
<p>Here are some of my favorite generated notes:</p>
|
||
|
||
<blockquote>
|
||
<p><em>“note: do not feel better”</em></p>
|
||
</blockquote>
|
||
|
||
<blockquote>
|
||
<p><em>“hi I am a car.”</em></p>
|
||
</blockquote>
|
||
|
||
<blockquote>
|
||
<p><em>“i am stuff and think about this before . this is it, the pond. how do they make me feel better?”</em></p>
|
||
</blockquote>
|
||
|
||
<blockquote>
|
||
<p><em>“i am still about the floor”</em></p>
|
||
</blockquote>
|
||
|
||
<p>Not perfect, but it is readable english, so i call it a win!</p>
|
||
|
||
<h2 id="play-with-the-code">Play with the code</h2>
|
||
<p>I have uploaded the basic code, the scraped posts, and a partial hdf5 file <a href="https://github.com/Ewpratten/be-bill">to GitHub</a> for anyone to play with. Maybe make a twitter bot out of this?</p></content><author><name></name></author><summary type="html">Bill Wurtz is an American musician who became reasonably famous through short musical videos posted to Vine and YouTube. I was searching through his website the other day, and stumbled upon a page labeled notebook, and thought I should check it out.</summary></entry><entry><title type="html">Building images from binary data</title><link href="http://0.0.0.0:4000/blog/2019/09/11/buildingimgfrombin" rel="alternate" type="text/html" title="Building images from binary data" /><published>2019-09-11T08:41:00-04:00</published><updated>2019-09-11T08:41:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/09/11/Buildingimgfrombin</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/09/11/buildingimgfrombin"><p>During a computer science class today, we were talking about embedding code and metadata in <em>jpg</em> and <em>bmp</em> files. @exvacuum was showing off a program he wrote that watched a directory for new image files, and would display them on a canvas. He then showed us a special image. In this image, he had injected some metadata into the last few pixels, which were not rendered, but told his program where to position the image on the canvas, and it’s size.</p>
|
||
|
||
<p>This demo got @hyperliskdev and I thinking about what else we can do with image data. After some talk, the idea of converting application binaries to images came up. I had seen a blog post about visually decoding <a href="https://en.wikipedia.org/wiki/On%E2%80%93off_keying">OOK data</a> by converting an <a href="http://www.ni.com/tutorial/4805/en/">IQ capture</a> to an image. With a little adaptation, I did the same for a few binaries on my laptop.</p>
|
||
|
||
<!-- Tweet embed -->
|
||
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I present: &quot;Parts of <a href="https://twitter.com/GIMP_Official?ref_src=twsrc%5Etfw">@GIMP_Official</a>&#39;s binary, represented as a bitmap&quot; <a href="https://t.co/iLljdE4nlK">pic.twitter.com/iLljdE4nlK</a></p>&mdash; Evan Pratten (@ewpratten) <a href="https://twitter.com/ewpratten/status/1171801959197794304?ref_src=twsrc%5Etfw">September 11, 2019</a></blockquote>
|
||
|
||
<h2 id="program-design">Program design</h2>
|
||
<p>Like all ideas I have, I wrote some code to test this idea out. Above is a small sample of the interesting designs found in the <a href="">gimp</a> binary. The goals for this script were to:</p>
|
||
|
||
<ul>
|
||
<li>Accept any file of any type or size</li>
|
||
<li>Allow the user to select the file dimensions</li>
|
||
<li>Generate an image</li>
|
||
<li>Write the data in a common image format</li>
|
||
</ul>
|
||
|
||
<p>If you would like to see how the code works, read “<em>check out the script</em>”.</p>
|
||
|
||
<h2 id="a-note-on-data-wrapping">A note on data wrapping</h2>
|
||
<p>By using a <a href="https://wiki.python.org/moin/Generators">generator</a>, and the <a href="https://docs.python.org/3/library/functions.html#func-range">range function</a>’s 3rd argument, any list can be easily split into a 2d list at a set interval.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Assuming l is a list of data, and n is an int that denotes the desired split location
|
||
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">l</span><span class="p">),</span> <span class="n">n</span><span class="p">):</span>
|
||
<span class="k">yield</span> <span class="n">l</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">i</span> <span class="o">+</span> <span class="n">n</span><span class="p">]</span>
|
||
</code></pre></div></div>
|
||
|
||
<h3 id="binaries-have-a-habit-of-not-being-rectangular">Binaries have a habit of not being rectangular</h3>
|
||
<p>Unlike photos, binaries are not generated from rectangular image sensors, but instead from compilers and assemblers (and sometimes hand-written binary). These do not generate perfect rectangles. Due to this, my script simply removes the last line from the image to “reshape” it. I may end up adding a small piece of code to pad the final line instead of stripping it in the future.</p>
|
||
|
||
<h2 id="other-file-types">Other file types</h2>
|
||
<p>I also looked at other file types. Binaries are very interesting because they follow very strict ordering rules. I was hoping that a <code class="highlighter-rouge">wav</code> file would do something similar, but that does not appear to be the case. This is the most interesting pattern I could find in a <code class="highlighter-rouge">wav</code> file:</p>
|
||
|
||
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Following up my previous post with a tiny segment of an audio file. This one is little less interesting <a href="https://t.co/u9EFloxnK5">pic.twitter.com/u9EFloxnK5</a></p>&mdash; Evan Pratten (@ewpratten) <a href="https://twitter.com/ewpratten/status/1171883910827040774?ref_src=twsrc%5Etfw">September 11, 2019</a></blockquote>
|
||
|
||
<p>Back to executable data, these are small segments of a <code class="highlighter-rouge">dll</code> file:</p>
|
||
|
||
<p><img src="/assets/images/dll.png" alt="Segment 1" /></p>
|
||
|
||
<p><img src="/assets/images/dll2.png" alt="Segment 2" /></p>
|
||
|
||
<h2 id="check-out-the-script">Check out the script</h2>
|
||
<p>This script is hosted <a href="https://github.com/Ewpratten/binmap">on my GitHub account</a> as a standalone file. Any version of python3 should work, but the following libraries are needed:</p>
|
||
|
||
<ul>
|
||
<li>Pillow</li>
|
||
<li>Numpy</li>
|
||
</ul></content><author><name></name></author><summary type="html">During a computer science class today, we were talking about embedding code and metadata in jpg and bmp files. @exvacuum was showing off a program he wrote that watched a directory for new image files, and would display them on a canvas. He then showed us a special image. In this image, he had injected some metadata into the last few pixels, which were not rendered, but told his program where to position the image on the canvas, and it’s size.</summary></entry><entry><title type="html">Doing Python OOP the wrong way</title><link href="http://0.0.0.0:4000/blog/2019/09/07/wrong-python" rel="alternate" type="text/html" title="Doing Python OOP the wrong way" /><published>2019-09-07T09:13:00-04:00</published><updated>2019-09-07T09:13:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/09/07/wrong-python</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/09/07/wrong-python"><p>If you know me, you probably know of the many weird things I do with python. Most recent of which being this <a href="https://en.wikipedia.org/wiki/Fizz_buzz">FizzBuzz</a> implementation in one line of python code:</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">_</span><span class="o">=</span><span class="p">[</span><span class="k">print</span><span class="p">(</span><span class="s">"FizzBuzz"</span><span class="p">[</span><span class="n">_</span><span class="o">*</span><span class="n">_</span><span class="o">%</span><span class="mi">3</span><span class="o">*</span><span class="mi">4</span><span class="p">:</span><span class="mi">8</span><span class="o">--</span><span class="n">_</span><span class="o">**</span><span class="mi">4</span><span class="o">%</span><span class="mi">5</span><span class="p">]</span> <span class="ow">or</span> <span class="n">_</span><span class="p">)</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">101</span><span class="p">)]</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>This installment of “weird things I do with python” will not focus on one-liners (that’s going on my todo list though). But instead, playing with Python’s classes and object system.</p>
|
||
|
||
<h2 id="a-quick-introduction-to-classes">A quick introduction to classes</h2>
|
||
<p>Im going to assume that you, the reader, have some reasonable knowledge of how computers work, and OOP concepts. If you do not, there are <a href="https://medium.com/swlh/5-free-object-oriented-programming-online-courses-for-programmers-156afd0a3a73">many great online resources</a> to help you out.</p>
|
||
|
||
<p>As a quick refresher, this is the Python syntax for a basic class:</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
|
||
|
||
<span class="c1"># This is the constructor. __init__ is an overridable python built-in
|
||
</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">arg1</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
|
||
|
||
<span class="c1"># Here we set the class' scoped my_number to arg1
|
||
</span> <span class="bp">self</span><span class="o">.</span><span class="n">my_number</span> <span class="o">=</span> <span class="n">arg1</span>
|
||
|
||
<span class="k">def</span> <span class="nf">printMyNumber</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">my_number</span><span class="p">)</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>This is really just a fancy setter and getter. Here is some example usage:</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_object</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
|
||
<span class="n">my_object</span><span class="o">.</span><span class="n">printMyNumber</span><span class="p">()</span> <span class="c1"># Prints 10
|
||
</span></code></pre></div></div>
|
||
|
||
<h2 id="noticing-something-odd">Noticing something odd</h2>
|
||
<p>Before reading the following, keep in mind that (as of now) I have not actually looked at the Python interpreter’s source code enough to know about their memory system. The following is just an educated guess.</p>
|
||
|
||
<p>Looking at any python class, you may notice that <strong>at least</strong> 1 argument is required. <code class="highlighter-rouge">self</code> is used to access the class’ data from itself. This is not present in most other languages I know, which means there might be something interesting happening behind the scenes. Here is a re-implementation of <code class="highlighter-rouge">MyClass</code> from above in java:</p>
|
||
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyClass</span> <span class="o">{</span>
|
||
<span class="kt">int</span> <span class="n">my_int</span><span class="o">;</span>
|
||
|
||
<span class="kd">public</span> <span class="nf">MyClass</span><span class="o">(</span><span class="kt">int</span> <span class="n">arg1</span><span class="o">){</span>
|
||
<span class="n">my_int</span> <span class="o">=</span> <span class="n">arg1</span><span class="o">;</span>
|
||
<span class="o">}</span>
|
||
|
||
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">printMyNumber</span><span class="o">(){</span>
|
||
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">my_int</span><span class="o">);</span>
|
||
<span class="o">}</span>
|
||
<span class="o">}</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>Notice the fact that there is no <code class="highlighter-rouge">self</code>? Yet Java methods can still access class data.</p>
|
||
|
||
<h2 id="implementing-objects-in-a-non-object-oriented-language">Implementing objects in a non-object oriented language</h2>
|
||
<p>In a non-OOP language (like C), objects can be faked by creating <a href="https://en.wikipedia.org/wiki/Struct_(C_programming_language)">structures</a> and some standard functions. These functions then take a pointer to their “parent” structure. Confusing? yes. But it works, and I see it used all over the place. Here a pseudocode example:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct MyClass {
|
||
int my_int; // Scpoed int
|
||
}
|
||
|
||
fn printMyNumber(MyClass* self){
|
||
print(self.my_int);
|
||
}
|
||
|
||
</code></pre></div></div>
|
||
|
||
<p><code class="highlighter-rouge">printMyNumber</code> takes a pointer to it’s “parent class”, called <code class="highlighter-rouge">self</code>. Look familiar? This is how Python works.</p>
|
||
|
||
<h2 id="lets-do-some-python">Let’s do some Python</h2>
|
||
<p>Alright.. Time for some “broken” Python. Here is yet another implementation of <code class="highlighter-rouge">MyClass</code>, except this time, each function is globally scoped:</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
|
||
<span class="c1"># Private, globally scoped functions
|
||
</span><span class="k">def</span> <span class="nf">_init_myclass</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">arg1</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">my_number</span> <span class="o">=</span> <span class="n">arg1</span>
|
||
|
||
<span class="k">def</span> <span class="nf">_myclass_printMyNumber</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">my_number</span><span class="p">)</span>
|
||
|
||
|
||
<span class="c1"># struct-like class containing function pointers
|
||
</span><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
|
||
|
||
<span class="n">__init__</span> <span class="o">=</span> <span class="n">_init_myclass</span>
|
||
<span class="n">printMyNumber</span> <span class="o">=</span> <span class="n">_myclass_printMyNumber</span>
|
||
|
||
</code></pre></div></div>
|
||
|
||
<p>This code will still function like a normal class. Unlike a regular class definition, the above code defines the constructor and <code class="highlighter-rouge">printMyNumber</code> methods in the global scope (marked as private with an underscore). A class is then created with function pointers to each of the global functions. This means that calling <code class="highlighter-rouge">MyClass.printMyNumber</code> will point to, and execute <code class="highlighter-rouge">_myclass_printMyNumber</code>. The interpreter still treats the underscore functions as members of <code class="highlighter-rouge">MyClass</code>, and passes the <code class="highlighter-rouge">self</code> argument along to them.</p>
|
||
|
||
<h2 id="why">Why?</h2>
|
||
<p>I have absolutely no idea why this would ever be useful. If you think you should start doing this in your code, <strong>don’t</strong>. It leads to very messy and confusing code, and is bad practice in just about every way.</p>
|
||
|
||
<p>The point of this post is to show yet another instance of the Python interpreter saying “<a href="https://www.urbandictionary.com/define.php?term=idgaf">idgaf</a>”, and letting us have a little fun.</p></content><author><name></name></author><summary type="html">If you know me, you probably know of the many weird things I do with python. Most recent of which being this FizzBuzz implementation in one line of python code: _=[print("FizzBuzz"[_*_%3*4:8--_**4%5] or _) for _ in range(101)]</summary></entry><entry><title type="html">I did some cleaning</title><link href="http://0.0.0.0:4000/blog/2019/08/27/github-cleanup" rel="alternate" type="text/html" title="I did some cleaning" /><published>2019-08-27T08:37:00-04:00</published><updated>2019-08-27T08:37:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/08/27/GitHub-cleanup</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/08/27/github-cleanup"><p>As I am continuing to check items off my TODO list before school starts, I have come to an item I have been putting off for a while. <strong>Clean up GitHub Account</strong>. Luckily, I discovered a little trick to make the process of deleting unused repos a little easier!</p>
|
||
|
||
<h2 id="getting-a-list-of-repos-to-delete">Getting a list of repos to delete</h2>
|
||
<p>I could have automated this, but I prefer a little control. To get the list, start by opening up a new Firefox window with a single tab. In this tab, open your GitHub profile to the list of repos.
|
||
Starting from the top, scroll through, and middle click on anything you want to delete. This opens it in a new tab.</p>
|
||
|
||
<p>Once you have a bunch of tabs open with repos to remove, use <a href="https://addons.mozilla.org/en-US/firefox/addon/urls-list/">this Firefox plugin</a> to create a plaintext list of every link you opened, and paste the list of links into VS-code.</p>
|
||
|
||
<h2 id="getting-an-api-token">Getting an API token</h2>
|
||
<p>Next, an API token is needed. Go to GitHub’s <a href="https://github.com/settings/tokens">token settings</a>, and generate a new one (make sure to enable repository deletion).</p>
|
||
|
||
<h2 id="parsing-the-links">“Parsing” the links</h2>
|
||
<p>With our new token, and out VS-code file, we can start “parsing” the data.</p>
|
||
|
||
<p>Pressing <code class="highlighter-rouge">CTRL + F</code> brings up the Find/Search toolbar. In the text box, there are a few icons. Pressing the one farthest to the right will enable <a href="https://en.wikipedia.org/wiki/Regular_expression">Regex</a> mode. With this set, paste the following:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://github.com/
|
||
</code></pre></div></div>
|
||
|
||
<p>Now, click the arrow on the left to enable <em>replace mode</em>, and put this in the new box:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl -XDELETE -H 'Authorization: token &lt;API token from above&gt;' "https://api.github.com/repos/
|
||
</code></pre></div></div>
|
||
|
||
<p>Then press <em>replace all</em>.</p>
|
||
|
||
<p>Finally, replace the contents of the first box with:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>\n
|
||
</code></pre></div></div>
|
||
|
||
<p>and the second with:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"\n
|
||
</code></pre></div></div>
|
||
|
||
<p>and <em>replace all</em> again.</p>
|
||
|
||
<h2 id="deleting-the-repos">Deleting the repos</h2>
|
||
<p>Simply copy the entire text file that was made, and paste it in a terminal, then press &lt;enter&gt; (this will take a while)</p></content><author><name></name></author><summary type="html">As I am continuing to check items off my TODO list before school starts, I have come to an item I have been putting off for a while. Clean up GitHub Account. Luckily, I discovered a little trick to make the process of deleting unused repos a little easier!</summary></entry><entry><title type="html">Keyed data encoding with Python</title><link href="http://0.0.0.0:4000/blog/2019/08/24/shift2" rel="alternate" type="text/html" title="Keyed data encoding with Python" /><published>2019-08-24T09:13:00-04:00</published><updated>2019-08-24T09:13:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/08/24/Shift2</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/08/24/shift2"><p>I have always been interested in text and data encoding, so last year, I made my first encoding tool. <a href="https://github.com/Ewpratten/shift64">Shift64</a> was designed to take plaintext data with a key, and convert it into a block of base64 that could, in theory, only be decoded with the original key. I had a lot of fun with this tool, and a very stripped down version of it actually ended up as a bonus question on the <a href="https://github.com/frc5024/Programming-Test/blob/master/test.md">5024 Programming Test</a> for 2018/2019. Yes, the key was in fact <code class="highlighter-rouge">5024</code>.</p>
|
||
|
||
<p>This tool had some issues. Firstly, the code was a mess and only accepted hard-coded values. This made it very impractical as an every-day tool, and a nightmare to continue developing. Secondly, the encoder made use of entropy bits, and self modifying keys that would end up producing encoded files &gt;1GB from just the word <em>hello</em>.</p>
|
||
|
||
<h2 id="shift2">Shift2</h2>
|
||
<p>One of the oldest items on my TODO list has been to rewrite shift64, so I made a brand new tool out of it. <a href="https://github.com/Ewpratten/shift">Shift2</a> is both a command-line tool, and a Python3 library that can efficiently encode and decode text data with a single key (unlike shift64, which used two keys concatenated into a single string, and separated by a colon).</p>
|
||
|
||
<h3 id="how-it-works">How it works</h3>
|
||
<p>Shift2 has two inputs. A <code class="highlighter-rouge">file</code>, and a <code class="highlighter-rouge">key</code>. These two strings are used to produce a single output, the <code class="highlighter-rouge">message</code>.</p>
|
||
|
||
<p>When encoding a file, shift2 starts by encoding the raw data with <a href="https://en.wikipedia.org/wiki/Ascii85">base85</a>, to ensure that all data being passed to the next stage can be represented as a UTF-8 string (even binary data). This base85 data is then XOR encrypted with a rotating key. This operation can be expressed with the following (this example ignores the base85 encoding steps):</p>
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">file</span> <span class="o">=</span> <span class="s">"Hello reader! I am some input that needs to be encoded"</span>
|
||
<span class="n">key</span> <span class="o">=</span> <span class="s">"ewpratten"</span>
|
||
|
||
<span class="n">message</span> <span class="o">=</span> <span class="s">""</span>
|
||
|
||
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">char</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="nb">file</span><span class="p">):</span>
|
||
<span class="n">message</span> <span class="o">+=</span> <span class="nb">chr</span><span class="p">(</span>
|
||
<span class="nb">ord</span><span class="p">(</span><span class="n">char</span><span class="p">)</span> <span class="o">^</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">])</span>
|
||
<span class="p">)</span>
|
||
|
||
</code></pre></div></div>
|
||
|
||
<p>The output of this contains non-displayable characters. A second base85 encoding is used to fix this. Running the example snippet above, then base85 encoding the <code class="highlighter-rouge">message</code> once results in:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CIA~89YF&gt;W1PTBJQBo*W6$nli7#$Zu9U2uI5my8n002}A3jh-XQWYCi2Ma|K9uW=@5di
|
||
</code></pre></div></div>
|
||
|
||
<p>If using the shift2 commandline tool, you would see a different output:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>B2-is8Y&amp;4!ED2H~Ix&lt;~LOCfn@P;xLjM_E8(awt`1YC&lt;SaOLbpaL^T!^W_ucF8Er~?NnC$&gt;e0@WAWn2bqc6M1yP+DqF4M_kSCp0uA5h-&gt;H
|
||
</code></pre></div></div>
|
||
|
||
<p>This is for a few reasons. Firstly, as mentioned above, shift2 uses base85 <strong>twice</strong>. Once before, and once after XOR encryption. Secondly, a file header is prepended to the output to help the decoder read the file. This header contains version info, the file length, and the encoding type.</p>
|
||
|
||
<h3 id="try-it-yourself-with-pip">Try it yourself with PIP</h3>
|
||
<p>I have published shift2 on <a href="https://pypi.org/project/shift-tool/">pypi.org</a> for use with PIP. To install shift2, ensure both <code class="highlighter-rouge">python3</code> and <code class="highlighter-rouge">python3-pip</code> are installed on your computer, then run:</p>
|
||
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install shift2</span>
|
||
pip3 <span class="nb">install </span>shift-tool
|
||
|
||
<span class="c"># View the help for shift2</span>
|
||
shift2 <span class="nt">-h</span>
|
||
</code></pre></div></div>
|
||
|
||
<div id="demo">
|
||
<h3 id="try-it-in-the-browser">Try it in the browser</h3>
|
||
<p>I have ported the core code from shift2 to <a href="http://www.brython.info/index.html">run in the browser</a>. This demo is entirely client-side, and may take a few seconds to load depending on your device.</p>
|
||
|
||
<p><input type="radio" id="encode" name="shift-action" value="encode" checked="" />
|
||
<label for="encode">Encode</label>
|
||
<input type="radio" id="decode" name="shift-action" value="decode" />
|
||
<label for="decode">Decode</label></p>
|
||
|
||
<p><input type="text" id="key" name="key" placeholder="Encoding key" required="" /><br />
|
||
<input type="text" id="msg" name="msg" placeholder="Message" required="" size="30" /></p>
|
||
|
||
<p><button type="button" class="btn btn-primary" id="shift-button" disabled="">shift2 demo is loading… (this may take a few seconds)</button></p>
|
||
|
||
</div>
|
||
|
||
<h3 id="future-plans">Future plans</h3>
|
||
<p>Due to the fact that shift2 can also be used as a library (as outlined in the <a href="https://github.com/Ewpratten/shift/blob/master/README.md">README</a>), I would like to write a program that allows users to talk to eachother IRC style over a TCP port. This program would use either a pre-shared, or generated key to encode / decode messages on the fly.</p>
|
||
|
||
<p>If you are interested in helping out, or taking on this idea for yourself, send me an email.</p>
|
||
|
||
<!-- Python code -->
|
||
<script type="text/python" src="/assets/python/shift2/shift2demo.py"></script></content><author><name></name></author><summary type="html">I have always been interested in text and data encoding, so last year, I made my first encoding tool. Shift64 was designed to take plaintext data with a key, and convert it into a block of base64 that could, in theory, only be decoded with the original key. I had a lot of fun with this tool, and a very stripped down version of it actually ended up as a bonus question on the 5024 Programming Test for 2018/2019. Yes, the key was in fact 5024.</summary></entry><entry><title type="html">How I set up ひらがな input on my laptop</title><link href="http://0.0.0.0:4000/blog/2019/08/12/setting-up-ja" rel="alternate" type="text/html" title="How I set up ひらがな input on my laptop" /><published>2019-08-12T15:40:00-04:00</published><updated>2019-08-12T15:40:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/08/12/Setting-up-JA</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/08/12/setting-up-ja"><p>I am currently working with <a href="https://en.wikipedia.org/wiki/Hiragana">ひらがな</a>, <a href="https://en.wikipedia.org/wiki/Katakana">かたかな</a>, and, <a href="https://en.wikipedia.org/wiki/Kanji">かんじ</a> in some projects, and needed a more reliable way to write than running some <a href="https://en.wikipedia.org/wiki/Romanization_of_Japanese">romaji</a> through an online translator. So, this post will detail what I did to enable native inputs on my laptop. This guide is specifically for <a href="https://i3wm.org/">i3wm</a>, because it does not obey system settings for languages and inputs.</p>
|
||
|
||
<h2 id="adding-font-support-to-linux">Adding font support to Linux</h2>
|
||
<p>Firstly, we need fonts. Depending on your system, these may already be installed. For Japanese, I only used <code class="highlighter-rouge">vlgothic</code>, so here in the package for it:</p>
|
||
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install fonts-vlgothic
|
||
</code></pre></div></div>
|
||
|
||
<h2 id="language-support">Language support</h2>
|
||
<p>Im not sure if this matters, but I have seen other people do it, so why not be safe?</p>
|
||
|
||
<p>I am currently running a stock Ubuntu <a href="">18.04</a> base, which means that everything is pre-configured for Gnome. To set language support in Gnome, pull up the settings panel:</p>
|
||
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># This line fixes some compatibility issues between </span>
|
||
<span class="c"># Gnome and I3 when launching the settings menu. </span>
|
||
<span class="c"># I recommend aliasing it.</span>
|
||
<span class="nb">env </span><span class="nv">XDG_CURRENT_DESKTOP</span><span class="o">=</span>GNOME gnome-control-center
|
||
</code></pre></div></div>
|
||
|
||
<p><img src="/assets/images/language-settings.png" alt="Gnome language settings" /></p>
|
||
|
||
<p>Next, go to <em>Settings &gt; Language and Region &gt; Input Sources</em>, and click on <em>Manage Installed Languages</em>.
|
||
This will bring up a window where you can select a new language to install. From here, I clicked on <em>Install / Remove Language</em>.</p>
|
||
|
||
<p><img src="/assets/images/language-installation.png" alt="Language installation panel" /></p>
|
||
|
||
<p>In this list, I just selected the languages I wanted (English and Japanese), and applied my changes. You may be asked to enter your password while installing the new languages. Once installation is complete, log out, and in again.</p>
|
||
|
||
<p>With the new language support installed, return to the <em>Input Sources</em> settings, and press the <code class="highlighter-rouge">+</code> button to add a new language. From here, search the language you want (it may be under <em>Other</em>) and select it. For Japanese, select the <code class="highlighter-rouge">mozc</code> variant.</p>
|
||
|
||
<p>Gnome’s language settings are now configured. If you are using Gnome (not I3), you can stop here.</p>
|
||
|
||
<h2 id="configuring-ibus">Configuring ibus</h2>
|
||
<p>Don’t get me wrong, I love I3wm, but sometimes it’s configurability drives me crazy.</p>
|
||
|
||
<p>After searching through various forums and wikis looking for an elegant way to switch languages in I3, I found a link to an <a href="https://wiki.archlinux.org/index.php/IBus">ArchWiki page</a> at the bottom of a mailing list (I blame Google for not showing this sooner). It turns out that a program called <code class="highlighter-rouge">ibus</code> is exactly what I needed. Here is how to set it up:</p>
|
||
|
||
<p>Remember <code class="highlighter-rouge">mozc</code> from above? If you are not using it, this package may not work. Search for the appropriate <code class="highlighter-rouge">ibus-</code> package for your selected language(s).</p>
|
||
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install ibus-mozc for Japanese (mozc)</span>
|
||
<span class="nb">sudo </span>apt <span class="nb">install </span>ibus-mozc
|
||
</code></pre></div></div>
|
||
|
||
<p>Now that <code class="highlighter-rouge">ibus</code> is installed, run the setup script:</p>
|
||
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ibus-setup
|
||
</code></pre></div></div>
|
||
|
||
<p><img src="/assets/images/ibus-general.png" alt="Ibus settings" /></p>
|
||
|
||
<p>From here, set your shortcut to something not used by I3 (I chose <code class="highlighter-rouge">CTRL+Shift+Space</code>, but most people prefer <code class="highlighter-rouge">Alt+Space</code>), and enable the system tray icon.
|
||
Now, go to the <em>Input Method</em> settings.</p>
|
||
|
||
<p><img src="/assets/images/ibus-input.png" alt="Ibus input settings" /></p>
|
||
|
||
<p>From here, press the <code class="highlighter-rouge">+</code>, and add your language(s).</p>
|
||
|
||
<h2 id="configuring-profile">Configuring .profile</h2>
|
||
<p>According to the Wiki page, I needed to add the following to my <code class="highlighter-rouge">~/.profile</code>:</p>
|
||
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Language support</span>
|
||
<span class="nb">export </span><span class="nv">GTK_IM_MODULE</span><span class="o">=</span>ibus
|
||
<span class="nb">export </span><span class="nv">XMODIFIERS</span><span class="o">=</span>@im<span class="o">=</span>ibus
|
||
<span class="nb">export </span><span class="nv">QT_IM_MODULE</span><span class="o">=</span>ibus
|
||
ibus-daemon <span class="nt">-d</span> <span class="nt">-x</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>It turns out that this <a href="https://github.com/ibus/ibus/issues/2020">causes issues with some browsers</a>, so I actually put <em>this</em> in my <code class="highlighter-rouge">~/.profile</code> instead:</p>
|
||
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Language support</span>
|
||
<span class="nb">export </span><span class="nv">GTK_IM_MODULE</span><span class="o">=</span>xim
|
||
<span class="nb">export </span><span class="nv">XMODIFIERS</span><span class="o">=</span>@im<span class="o">=</span>ibus
|
||
<span class="nb">export </span><span class="nv">QT_IM_MODULE</span><span class="o">=</span>xim
|
||
ibus-daemon <span class="nt">-drx</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>Now, log out and in again to let ibus properly start again, and there should now be a new applet in your bar for language settings.</p>
|
||
|
||
<h2 id="workflow">Workflow</h2>
|
||
<p><code class="highlighter-rouge">ibus</code> runs in the background and will show an indication of your selected language upon pressing the keyboard shortcut set in the <a href="#configuring-ibus">setup tool</a>. For languages like Japanese, where it’s writing systems do not use the English / Latin-based alphabets, <code class="highlighter-rouge">ibus</code> will automatically convert your words as you type (this behavior will be different from language to language).</p>
|
||
|
||
<p>An example of this is as follows. I want to write the word <em>Computer</em> in Japanese (Katakana to be exact). I would switch to <code class="highlighter-rouge">mozc</code> input, and start typing the romaji word for computer, <em>Pasokon</em>. This will automatically be converted to Hiragana, <em>ぱそこん</em>. <em>Computer</em> is not a word that one would write in Hiragana as far as I know, so Katakana would be a better choice. To convert this word, I just press <code class="highlighter-rouge">Space</code> (This is indicated in the bottom left of my screen by <code class="highlighter-rouge">ibus</code>), and I now have <em>パソコン</em>, the Katakana word for <em>Computer</em>!</p>
|
||
|
||
<hr />
|
||
|
||
<h4 id="after-note-languages">After Note: Languages</h4>
|
||
<p>In case you can’t tell, English is my native language. If I messed up my spelling or context with the small amount of Japanese in this post, <a href="/about#chat-with-me">let me know</a>!</p></content><author><name></name></author><summary type="html">I am currently working with ひらがな, かたかな, and, かんじ in some projects, and needed a more reliable way to write than running some romaji through an online translator. So, this post will detail what I did to enable native inputs on my laptop. This guide is specifically for i3wm, because it does not obey system settings for languages and inputs.</summary></entry><entry><title type="html">My weird piece of EDC</title><link href="http://0.0.0.0:4000/blog/2019/08/10/why-i-carry-nfc" rel="alternate" type="text/html" title="My weird piece of EDC" /><published>2019-08-10T16:57:00-04:00</published><updated>2019-08-10T16:57:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/08/10/Why-I-Carry-NFC</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/08/10/why-i-carry-nfc"><p>Im back with a quick little post about something I cary with me everywhere I go, EDC (Every-Day Carry) if you will.</p>
|
||
|
||
<h2 id="how-this-started">How this started</h2>
|
||
<p>Earlier this year, my friend @hyperliskdev showed me a piece of “fake ID” he was given as a joke. After some experimentation, he noticed that, upon tapping it to his phone, he would get an error message about an un-formatted card.</p>
|
||
|
||
<p>After hearing of this, I opened up <a href="https://play.google.com/store/apps/details?id=com.wakdev.nfctools.pro">NFC Tools</a> on my phone and started playing. We had quite some fun with <a href="#shenanigans">various settings and data</a>, and I decided that I wanted a card too. I send a message to someone that I knew worked with these, and got myself 4 to play with.</p>
|
||
|
||
<h2 id="shenanigans">Shenanigans</h2>
|
||
<p>Upon figuring out how to write to @hyperliskdev’s card, we started out simple. We sent bits of text to eachother, and I eventually sent him a copy of my contact information, and bitcoin address. Then, came the real fun..</p>
|
||
|
||
<p>By setting the data type to <code class="highlighter-rouge">external link</code>, and the content to <a href="https://www.youtube.com/watch?v=dQw4w9WgXcQ">this totally not suspicious URL</a>, we now had the perfect tool for derailing a lesson. An automatic <a href="https://en.wikipedia.org/wiki/Rickrolling">Rick Roll</a> card. Upon tapping this card to a phone, the youtube app would auto-play <em>Rick Astley’s Never Gonna Give You Up</em>. After this discovery, people started asking to buy pre-configured cards from me :laughing:.</p>
|
||
|
||
<p>After this came even more fun ideas:</p>
|
||
<ul>
|
||
<li>Enabling flashlights</li>
|
||
<li>Rebooting phones</li>
|
||
<li>Calling phone numbers</li>
|
||
<li>Sending texts</li>
|
||
<li>Filling phones with fake contacts</li>
|
||
</ul>
|
||
|
||
<h2 id="practical-use">Practical use</h2>
|
||
<p>I don’t actually carry my cards around for messing with people but instead, use them for things like:</p>
|
||
<ul>
|
||
<li>Cloning hotel access cards (being in a room of 4 with only 2 cards)</li>
|
||
<li>Creating login cards for school printers (so I don’t have to log in manually)</li>
|
||
<li>Sharing small amounts of data and links between phones</li>
|
||
<li>Giving my contact info to people</li>
|
||
</ul>
|
||
|
||
<p>Thanks to the NFC Tools app, pretty much everything is 3 taps and a swipe away. I strongly recommend picking up some cards for yourself if wou work with a large number of NFC-compatible systems.</p>
|
||
|
||
<h2 id="an">A/N</h2>
|
||
<p>Occasionally, I either have nothing in the works, or am working on some very boring and technical projects, so I look to post some fun content like this. Currently the latter of the options is true, and I wanted a quick break from writing networking code.</p>
|
||
|
||
<p>Let me know what you think of this type of content!</p></content><author><name></name></author><summary type="html">Im back with a quick little post about something I cary with me everywhere I go, EDC (Every-Day Carry) if you will.</summary></entry><entry><title type="html">Mind map generation with Python</title><link href="http://0.0.0.0:4000/blog/2019/07/15/mindmap" rel="alternate" type="text/html" title="Mind map generation with Python" /><published>2019-07-15T14:38:00-04:00</published><updated>2019-07-15T14:38:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/07/15/MindMap</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/07/15/mindmap"><p>While working on an assignment with <a href="https://coggle.it">Coggle</a> today, I noticed an interesting option in the save menu. <em>Download as .mm file</em>. Having rarely worked with mind maps before, and only doing it online, it never occured to me that someone would have a file format for it. So I took a look.</p>
|
||
|
||
<h2 id="what-is-a-mm-file">What is a .mm file?</h2>
|
||
<p>It turns out, a <code class="highlighter-rouge">.mm</code> file is just some XML describing the mind map. Here is a simple mind map:</p>
|
||
|
||
<p><img src="/assets/images/mindmap-simple.png" alt="Simple Mind Map" /></p>
|
||
|
||
<p>And again as a <code class="highlighter-rouge">.mm</code> file:</p>
|
||
|
||
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;map</span> <span class="na">version=</span><span class="s">"0.9.0"</span><span class="nt">&gt;</span>
|
||
<span class="nt">&lt;node</span> <span class="na">TEXT=</span><span class="s">"Master Node"</span> <span class="na">FOLDED=</span><span class="s">"false"</span> <span class="na">POSITION=</span><span class="s">"right"</span> <span class="na">ID=</span><span class="s">"5d2d02b1a315dd0879f48c1c"</span> <span class="na">X_COGGLE_POSX=</span><span class="s">"0"</span> <span class="na">X_COGGLE_POSY=</span><span class="s">"0"</span><span class="nt">&gt;</span>
|
||
<span class="nt">&lt;edge</span> <span class="na">COLOR=</span><span class="s">"#b4b4b4"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;font</span> <span class="na">NAME=</span><span class="s">"Helvetica"</span> <span class="na">SIZE=</span><span class="s">"17"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;node</span> <span class="na">TEXT=</span><span class="s">"Child branch"</span> <span class="na">FOLDED=</span><span class="s">"false"</span> <span class="na">POSITION=</span><span class="s">"right"</span> <span class="na">ID=</span><span class="s">"f72704969525d2a0333dd635"</span><span class="nt">&gt;</span>
|
||
<span class="nt">&lt;edge</span> <span class="na">COLOR=</span><span class="s">"#7aa3e5"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;font</span> <span class="na">NAME=</span><span class="s">"Helvetica"</span> <span class="na">SIZE=</span><span class="s">"15"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;node</span> <span class="na">TEXT=</span><span class="s">"Children 1"</span> <span class="na">FOLDED=</span><span class="s">"false"</span> <span class="na">POSITION=</span><span class="s">"right"</span> <span class="na">ID=</span><span class="s">"c83826af506cae6e55761d5c"</span><span class="nt">&gt;</span>
|
||
<span class="nt">&lt;edge</span> <span class="na">COLOR=</span><span class="s">"#7ea7e5"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;font</span> <span class="na">NAME=</span><span class="s">"Helvetica"</span> <span class="na">SIZE=</span><span class="s">"13"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;/node&gt;</span>
|
||
<span class="nt">&lt;node</span> <span class="na">TEXT=</span><span class="s">"Children 2"</span> <span class="na">FOLDED=</span><span class="s">"false"</span> <span class="na">POSITION=</span><span class="s">"right"</span> <span class="na">ID=</span><span class="s">"47723a4d0fb766863f70d204"</span><span class="nt">&gt;</span>
|
||
<span class="nt">&lt;edge</span> <span class="na">COLOR=</span><span class="s">"#82aae7"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;font</span> <span class="na">NAME=</span><span class="s">"Helvetica"</span> <span class="na">SIZE=</span><span class="s">"13"</span><span class="nt">/&gt;</span>
|
||
<span class="nt">&lt;/node&gt;</span>
|
||
<span class="nt">&lt;/node&gt;</span>
|
||
<span class="nt">&lt;/node&gt;</span>
|
||
<span class="nt">&lt;/map&gt;</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>Neat, right?</p>
|
||
|
||
<h2 id="what-can-we-do-with-it">What can we do with it?</h2>
|
||
<p>I have not done much research about this because I wanted to work all of this out on my own. But I know one thing as a fact: working with XML sucks (especially in Python). I decided that this would be much better if I could load <code class="highlighter-rouge">.mm</code> files as JSON. This would allow easy manipulation and some cool projects.</p>
|
||
|
||
<h2 id="my-script">My script</h2>
|
||
<p>Like everything I do, I made a script to play with these files.</p>
|
||
|
||
<p>It’s pretty simple. First, It loads a <code class="highlighter-rouge">.mm</code> file, then parses it into a <code class="highlighter-rouge">list</code> of <code class="highlighter-rouge">xml.etree.ElementTree.Element</code>.</p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">raw_mm</span> <span class="o">=</span> <span class="s">""</span>
|
||
|
||
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="nb">file</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
|
||
<span class="n">raw_mm</span> <span class="o">=</span> <span class="n">fp</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
||
<span class="n">fp</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
|
||
<span class="n">xml</span> <span class="o">=</span> <span class="n">ET</span><span class="o">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">raw_mm</span><span class="p">)</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>The parsed <code class="highlighter-rouge">list</code> is then passed into a recursive function that constructs a <code class="highlighter-rouge">dict</code></p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">xmlToDict</span><span class="p">(</span><span class="n">xml</span><span class="p">):</span>
|
||
<span class="n">output</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="k">for</span> <span class="n">elem</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">xml</span><span class="p">):</span>
|
||
|
||
<span class="k">if</span> <span class="s">"TEXT"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">elem</span><span class="o">.</span><span class="n">attrib</span><span class="p">:</span>
|
||
<span class="k">continue</span>
|
||
|
||
<span class="n">name</span> <span class="o">=</span> <span class="n">elem</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">'TEXT'</span><span class="p">]</span>
|
||
<span class="n">json_element</span> <span class="o">=</span> <span class="p">{</span><span class="s">"name"</span><span class="p">:</span> <span class="n">name</span><span class="p">}</span>
|
||
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">json_element</span><span class="p">[</span><span class="s">"children"</span><span class="p">]</span> <span class="o">=</span> <span class="n">xmlToDict</span><span class="p">(</span><span class="n">elem</span><span class="p">)</span>
|
||
<span class="k">except</span><span class="p">:</span>
|
||
<span class="k">continue</span>
|
||
|
||
<span class="c1"># Detect node type
|
||
</span> <span class="k">if</span> <span class="n">json_element</span><span class="p">[</span><span class="s">"children"</span><span class="p">]:</span>
|
||
<span class="n">json_element</span><span class="p">[</span><span class="s">"type"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"branch"</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="n">json_element</span><span class="p">[</span><span class="s">"type"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"leaf"</span>
|
||
<span class="k">del</span> <span class="n">json_element</span><span class="p">[</span><span class="s">"children"</span><span class="p">]</span>
|
||
|
||
<span class="n">output</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">json_element</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">output</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>Finally, the <code class="highlighter-rouge">dict</code> is written to a file with <code class="highlighter-rouge">json.dump</code></p>
|
||
|
||
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">json</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">mind_map</span><span class="p">,</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="nb">file</span> <span class="o">+</span> <span class="s">".json"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">))</span>
|
||
</code></pre></div></div>
|
||
|
||
<p>The whole script (with comments) can be found on my <a href="https://gist.github.com/Ewpratten/0d8f7c7371380c9ca8adcfc6502ccf84#file-parser-py">GitHub account</a>.</p>
|
||
|
||
<h2 id="the-output">The output</h2>
|
||
<p>Running the <code class="highlighter-rouge">.mm</code> file from above through the script gives:</p>
|
||
|
||
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
|
||
</span><span class="p">{</span><span class="w">
|
||
</span><span class="nl">"name"</span><span class="p">:</span><span class="s2">"Master Node"</span><span class="p">,</span><span class="w">
|
||
</span><span class="nl">"children"</span><span class="p">:[</span><span class="w">
|
||
</span><span class="p">{</span><span class="w">
|
||
</span><span class="nl">"name"</span><span class="p">:</span><span class="s2">"Child branch"</span><span class="p">,</span><span class="w">
|
||
</span><span class="nl">"children"</span><span class="p">:[</span><span class="w">
|
||
</span><span class="p">{</span><span class="w">
|
||
</span><span class="nl">"name"</span><span class="p">:</span><span class="s2">"Children 1"</span><span class="p">,</span><span class="w">
|
||
</span><span class="nl">"type"</span><span class="p">:</span><span class="s2">"leaf"</span><span class="w">
|
||
</span><span class="p">},</span><span class="w">
|
||
</span><span class="p">{</span><span class="w">
|
||
</span><span class="nl">"name"</span><span class="p">:</span><span class="s2">"Children 2"</span><span class="p">,</span><span class="w">
|
||
</span><span class="nl">"type"</span><span class="p">:</span><span class="s2">"leaf"</span><span class="w">
|
||
</span><span class="p">}</span><span class="w">
|
||
</span><span class="p">],</span><span class="w">
|
||
</span><span class="nl">"type"</span><span class="p">:</span><span class="s2">"branch"</span><span class="w">
|
||
</span><span class="p">}</span><span class="w">
|
||
</span><span class="p">],</span><span class="w">
|
||
</span><span class="nl">"type"</span><span class="p">:</span><span class="s2">"branch"</span><span class="w">
|
||
</span><span class="p">}</span><span class="w">
|
||
</span><span class="p">]</span><span class="w">
|
||
</span></code></pre></div></div>
|
||
|
||
<h2 id="the-next-step">The next step</h2>
|
||
<p>This script just translates a <code class="highlighter-rouge">.mm</code> file to JSON. Nothing else. Next, I want to convert this to a library, and add a JSON to <code class="highlighter-rouge">.mm</code> function as well. This leads into my ultimate goal for this project.</p>
|
||
|
||
<p>I want a script that I can drop in the root of any project to build a <a href="https://gource.io/">Gource</a>-style visualization of the folder structure. This will give me a way to make cool visualizations for lessons on the robotics team. On top of the folder visualization, Coggle’s new flowchart feature can be used to generate graphical representations of @frc5024’s codebases. This could give me an interactive overview of the work being done by our team.</p>
|
||
|
||
<h3 id="further-learning">Further learning</h3>
|
||
<p>crm.org has done a great writeup of <a href="https://crm.org/news/free-flowin-mind-maps-with-coggle">Coggle, and some of it’s features</a>. If you are looking to learn more about the tool, I recommend taking a few minute to read their post.</p></content><author><name></name></author><summary type="html">While working on an assignment with Coggle today, I noticed an interesting option in the save menu. Download as .mm file. Having rarely worked with mind maps before, and only doing it online, it never occured to me that someone would have a file format for it. So I took a look.</summary></entry><entry><title type="html">Taking a look back at GMAD</title><link href="http://0.0.0.0:4000/blog/2019/07/13/lookback-gmad" rel="alternate" type="text/html" title="Taking a look back at GMAD" /><published>2019-07-13T10:43:00-04:00</published><updated>2019-07-13T10:43:00-04:00</updated><id>http://0.0.0.0:4000/blog/2019/07/13/Lookback-GMAD</id><content type="html" xml:base="http://0.0.0.0:4000/blog/2019/07/13/lookback-gmad"><p>One day, back in June of 2018, I was both looking for a new project to work on, and trying to decide which Linux distro to install on one of my computers. From this, a little project was born. <a href="/gmad">Give Me a Distro</a> (or, GMAD, as I like to call it) is a little website that chooses a random distribution of Linux and shows a description of what you are about to get yourself into, and a download link for the latest ISO.</p>
|
||
|
||
<h2 id="backend-tech">Backend tech</h2>
|
||
<p>This is one of the simplest projects I have ever made. All the backend does is:</p>
|
||
<ul>
|
||
<li>Select a random number (n)</li>
|
||
<li>Fetch the nth item from a list of distros</li>
|
||
<li>Push the selected data to the user via DOM</li>
|
||
</ul>
|
||
|
||
<h2 id="frontend">Frontend</h2>
|
||
<p>This website is just plain HTML and CSS3, built without any CSS framework.</p>
|
||
|
||
<h2 id="my-regrets">My regrets</h2>
|
||
<p>There are two things I do not like about this project. Firstly, on load, the site breifly suggests Arch Linux before flashing to the random selection. This is due to the fact that Arch is the default for people with Javascript disabled. Some kind of loading animation would fix this.</p>
|
||
|
||
<p>Secondly, the version of the site hosted on <a href="https://retrylife.ca/gmad">retrylife.ca</a> is actually just an iframe to <a href="https://ewpratten.github.io/GiveMeADistro">ewpratten.github.io</a> due to some CNAME issues.</p>
|
||
|
||
<h2 id="contributing">Contributing</h2>
|
||
<p>If you would like to add a distro or three to the website, feel free to make a pull request over on <a href="https://github.com/Ewpratten/GiveMeADistro">GitHub</a>.</p>
|
||
|
||
<h2 id="why-make-a-post-about-it-a-year-later">Why make a post about it a year later?</h2>
|
||
<p>I just really enjoyed working with the project and sharing it with friends, so I figured I should mention it here too. Maybe it will inspire someone to make something cool!</p></content><author><name></name></author><summary type="html">One day, back in June of 2018, I was both looking for a new project to work on, and trying to decide which Linux distro to install on one of my computers. From this, a little project was born. Give Me a Distro (or, GMAD, as I like to call it) is a little website that chooses a random distribution of Linux and shows a description of what you are about to get yourself into, and a download link for the latest ISO.</summary></entry></feed> |