{{Header}}
{{#seo:
|description=Checking {{project_name_long}} for backdoors.
|image=Backdoor-23423523523.jpg
}}
{{intro|
Checking {{project_name_short}} for backdoors.
}}
{{archived}}
= Manual Builds vs Build Script Builds =
In oversimplified terms, {{project_name_short}} is just a collection of configuration files and scripts.

* Manual Builds: One could start with Debian (or any Linux distribution) then use mice and keyboard to re-configure it to become something similar to {{project_name_short}}.
* Build Script Builds: Such builds are created using a build script without any manual developer modifications as soon as the build script was started. All source code has been published prior or running the build script. All steps required such as building packages and creating a binary image are done automated by a build script.

Only build script builds are permitted for redistribution as per {{project_name_short}} policy.

Redistribution of manual builds would be unprofessional.

Advantages of script created builds are that these are "cleaner". For a list of advantages, see [[Dev/Installation_from_Repository#Distro_Morphing_vs_Builds|this chapter]].

= Verifiable Builds =
== Verifiable .ova's ==
=== Introduction - What this Achieves ===
{{Verifiable_Ovas_Introduction}}

=== Short Overview - How it Works ===
When building {{project_name_short}} {{Code2|.ova}}'s from source code, for example {{Code2|/home/user/whonix_binary/{{project_name_gateway_short}}-9.ova}} also report {{Code2|/home/user/whonix_binary/{{project_name_gateway_short}}-9.report}} file will be created.

After building the {{Code2|.ova}}, the last build-step will extract the {{Code2|.ova}}, which will result in getting three files, i.e. {{Code2|{{project_name_gateway_short}}-9.ovf}}, {{Code2|{{project_name_gateway_short}}-9.mf}} and {{Code2|{{project_name_gateway_short}}-9-disk1.vmdk}}. The {{Code2|.vmdk}} <ref>
https://en.wikipedia.org/wiki/VMDK
</ref> image will then be converted to {{Code2|.vdi}} <ref>
https://en.wikipedia.org/wiki/VDI_(file_format)
</ref>, then converted to {{Code2|raw}} <ref>
https://en.wikipedia.org/wiki/Raw_image_format
</ref> ({{Code2|.img}}. (Converting as in creating a copy, not actually converting.) The filesystem layout, the MBR <ref>
https://en.wikipedia.org/wiki/Master_boot_record
</ref> and the VBR <ref>
https://en.wikipedia.org/wiki/Volume_boot_record
</ref> are written into separate files. The raw image will then be mounted. A sha512 checksum of every file will be created and stored in the report.

Anyone building {{Code2|{{project_name_short}}.ova}} images will hopefully end up with the same report. We can then compare the reports using tools such as {{Code2|diff}} and/or {{Code2|meld}}. Those reports should be very similar. We can't end up with the very same reports, because again, there are no deterministically built operating systems yet. However, we can manually review the few remaining differences. That should make it reasonable to believe, that the original {{Code2|{{project_name_short}}.ova}} images have been built from the source code that has been published for that {{project_name_short}} version.

Below on this page is a longer How-To.

=== Current Limitations ===
During the build process of {{project_name_gateway_long}} we are still using torproject.org's Tor repository for Debian testing <ref>
https://deb.torproject.org/torproject.org/dists/testing/main/binary-i386/Packages
</ref>. Should that version get upgraded, this will create differences. In theory, old files could still be verified against torproject's signing keys, but that would require them to be archived somewhere. [https://tor.stackexchange.com/questions/990/is-there-an-archive-of-deb-torproject-org/1000?noredirect=1#1000 There is currently no such archive. (Peter Palfrader offered to provide the old packages on request).] {{project_name_short}} is currently only easily verifiable as long as torproject.org's repository does not get upgraded. In future, when Tor 0.2.4 migrates into Debian testing, this limitation may be lifted. We're happy to hear better solutions in meanwhile.

Our {{Code2|[https://github.com/adrelanos/{{project_name_short}}/blob/master/help-steps/analyze_image help-steps/analyze_image]}} script does not check the contents of the image outside the file system. We're not aware of any method getting a backdoor activated by writing malicious data into those places. We might get rid of this limitation in future, see chapter Possible Future Improvements.

We don't check GUID <ref>
https://en.wikipedia.org/wiki/GUID_Partition_Table
</ref>, because we're not using GUID. <ref>
You can verify that by auditing {{project_name_short}} source code.
</ref>

=== Attack Surface ===
The {{Code2|/help-steps/analyze_image}} script is supposed to execute no code from the image you are analyzing. The risk running that script on images you did not create yourself such as on the official images should be low, but not non-existent. Audits of the analyzing script itself are most welcome. In order to analyze the image, tools which are available in Debian's repository are used. Such as extracting the .ova requires {{Code2|tar}}<ref>
https://packages.debian.org/{{Stable project version based on Debian codename}}/tar
</ref>. In theory, the {{Code2|.ova}} could be malformed in a way to exploit the auditor's machine while extracting it to coerce the auditor's machine create a legit report hiding the actual backdoor (and infecting the auditor's machine with a backdoor). To find out what other tools are used and for what, please have a look at the {{Code2|/help-steps/analyze_image}} script.

=== Theoretical Holes ===
Unfortunately, we can not directly mount the {{Code2|.ova}}. We have to extract it first. Rather, VirtualBox does not support creating {{Code2|.ova}} containing {{Code2|.img}} (raw images). Img's would be ideal, since low level tools such as {{Code2|dd}} understand them. Even worse, {{Code2|.ova}}'s created by VirtualBox always include {{Code2|.vmdk}}. [https://forums.virtualbox.org/viewtopic.php?f=1&t=58460 export VM using VDI instead of VMDK is not possible]. Rather [https://forums.virtualbox.org/viewtopic.php?f=7&t=58461 Mounting VMDK's on Debian testing/{{Stable project version based on Debian codename}}] using Free Software is currently not possible since no software supports that. During the convert process {{Code2|ova &rarr; vmdk &rarr; vdi &rarr; img}}, the MBR, bootloader, VBR and/or other obscure differences could occur. There is no reason to believe that is actually the case, but we're not aware of any research on that topic.

=== Your Own Alternatives ===
The {{Code2|/help-steps/analyze_image}} script is the tool which comes with {{project_name_short}} source code to audit {{Code2|{{project_name_short}}.ova}}. Nothing prevents you from cooking up your own custom method to verify {{Code2|{{project_name_short}}.ova}}. On the contrary, we'd be very happy to hear about your methods, source code and results. Since this hasn't ever happened in past, {{project_name_short}} came up with its own {{Code2|/help-steps/analyze_image}} script.

=== Possible Future Improvements ===
==== sleuthkit ====
[https://github.com/sleuthkit/sleuthkit/ sleuthkit] could be an interesting addition or even replacement for {{project_name_short}} custom script. Then something like this could perhaps be used.

<pre>
mmls ~/whonix_binary/{{project_name_gateway_short}}-9.raw
</pre>

<pre>
fsstat -o 0000004096 ~/whonix_binary/{{project_name_gateway_short}}-9.raw
</pre>

<pre>
fls -r -o 0000004096 ~/whonix_binary/{{project_name_gateway_short}}-9.raw
</pre>

<pre>
tsk_gettimes ~/whonix_binary/{{project_name_gateway_short}}-9.raw
</pre>

<pre>
fiwalk ~/whonix_binary/{{project_name_gateway_short}}-9.raw
</pre>

Do these tools produce very similar results when run on very similar images or does it mess up the ordering?

fiwalk unfortunately only supports md5 and sha1 and [https://github.com/sleuthkit/sleuthkit/issues/182 not yet SHA-256 and/or SHA-3].

<ref>
[https://github.com/sleuthkit/sleuthkit/issues/354 (Solved!) fls / fstat issue: Invalid magic value (not an EXTxFS file system (magic))]
</ref>

==== Issue Tracker ====
https://phabricator.whonix.org/tag/verifiable_builds/

=== Expected Differences ===
* {{Code|(mount_folder) /boot/initrd.img*}} - Those are extracted to the {{Code|initrd_folder}} and diffed.
* {{Code|(mount_folder) /etc/shadow}} - Manually look at the diff. Users are advised to change their passwords anyway.
* {{Code|(mount_folder) /etc/shadow-}} - Same as above.
* {{Code|(mount_folder) /etc/init.d/.depend.boot}} - {{Code|/sbin/insserv}} (part of sysvinit) does not produce deterministic results. Gets copied to the {{Code|manual_analysis_folder}}. Manually review them. You just have to check, they're not containing anything sketchy. You most likely won't be able to tell, if some services are deliberately not started or in wrong order. This is not an issue, because when {{project_name_short}} starts for the first time it executes {{Code|whonix_shared/usr/lib/whonix/first_run_initializer}}, which executes {{Code|/sbin/insserv}}, which recreates this file. <ref>
We can not delete this file before redistributing {{project_name_short}}, because then {{project_name_short}} wouldn't start.
</ref>
* {{Code|(mount_folder) /etc/init.d/.depend.start}} - Same as above.
* {{Code|(mount_folder) /etc/init.d/.depend.stop}} - Same as above.
* {{Code|(mount_folder) /usr/share/whonix/build_timestamp}} - Timestamp, naturally differs.
* {{Code|(mount_folder) /var/lib/initramfs-tools/...}} - Is a one-line text file, which contains the name of the initrd and its checksum. Differs, because the initrd is not deterministic in the first place.
* {{Code|(mount_folder) /var/lib/initramfs-tools/...}} - Same as above.

=== Unexpected Differences ===
There could be differences besides the expected differences, because this is still experimental as in a new feature. Remember, there are no deterministically built distributions yet. If you find any additional differences, keep calm, it is not necessarily a backdoor, report them, and if you can, diff and/or audit them.

=== Folder Overview ===
Your report root folder can most likely be found in {{Code|/home/user/whonix_binary/$VMNAME-${derivative-maker_whonix_version_new}_report_tempfolder}}.

{{Code|extracted_ova_folder}} - Should contain 3 files. One {{Code2|.vmdk}}, which you can ignore, that's what we unpack - if we could diff it in the first place we wouldn't have so much trouble. Since the contents of the extracted_ova_folder gets added to the file_list, it won't be possible, that there are any extra files - if there were, you could see them in the report and should be suspicious.

{{Code|vdi_folder}} - Should only contain one {{Code2|.vdi}}. If we could diff it in the first place we wouldn't have so much trouble.

{{Code|raw_folder}} - Should only contain one {{Code2|.img}}. Same as above.

{{Code|auto_hash_folder}} - Are all added to the {{Code2|.report}} and {{Code2|file_list}} and should contain no differences.

{{Code|extracted_initrd_folder}} - The are the extracted {{Code|(mount_folder) /boot/initrd.img*}} files. Are all added to the {{Code2|.report}} and {{Code2|file_list}} and should contain no differences.

{{Code|manual_analysis_folder}} - Not hashed in the {{Code2|.report}} file, because those will show differences anyway which we expect. Those have to be manually audited.

{{Code|debug_folder}} - We copy these files out of the images just in case for convenience. Those have been already checksum'ed (in the {{Code2|.report}} file) as regular files within the (mount_folder). Feel free to ignore that folder. It is only in for debugging purposes.

=== How-To ===
First of all, you need to create your own {{project_name_short}}.ova images from source code, refer to [[Dev/Build Documentation|Build Documentation]]. Your report file can most likely be found at {{Code|/home/user/whonix_binary/$VMNAME-$derivative-maker_whonix_version_new.report}} and your report_tempfolder at {{Code|/home/user/whonix_binary/$VMNAME-${derivative-maker_whonix_version_new}_tempfolder}}. That is your own analyzed {{Code2|.ova}}. To make space for another analysis for other builds (such as those from whonix.org), rename your whonix_binary folder to something else, such as {{Code2|whonix_binary1}}.

If you've built for example {{project_name_short}} 9, you have to download {{project_name_short}} 9 as well. Place the {{Code2|.ova}} in {{Code|/home/user/whonix_binary/}}. Keep the original name. Then go to {{project_name_short}} source code folder ({{Code|cd /home/user/{{project_name_short}}}} and analyze that image. Use.

<pre>
## {{project_name_short}} 9
sudo ./build-steps.d/5100_create-report --tor-gateway

## {{project_name_short}} 13/14
sudo ./build-steps.d/5100_create-report --build --flavor whonix-gateway
</pre>

(And later repeat that step with the --tor-workstation switch.)

Then you have two {{Code2|.report}} files and two report temp folders.

Use your favorite diff viewer for huge files. For example {{Code2|meld}}. Diff the reports. For example, use.

<pre>
meld ./whonix_binary1/{{project_name_gateway_short}}-9.report ./whonix_binary/{{project_name_gateway_short}}-9.report
</pre>

These files should be very similar, expect for a few differences which we earlier mentioned in chapter Expected Differences. Look at the differences. Two of them (because we're installing two kernels) should look like this:

{{Code|(mount_folder) /boot/initrd.img...}}

Check at the top of your report files for (extracted_initrd_folder) and see if both initrd's were really extracted. <ref>
They should, because otherwise if extraction failed, the {{Code2|analyze_image}} script should have failed earlier and informed you already.
</ref>

Also ignore {{Code|(mount_folder) /etc/init.d/.depend.boot}}, {{Code|(mount_folder) /etc/init.d/.depend.start}}, and {{Code|(mount_folder) /etc/init.d/.depend.stop}} for now, this is an expected difference and we audit them later.

Also ignore {{Code|(mount_folder) /etc/shadow}} and {{Code|(mount_folder) /etc/shadow-}} for now, this is an expected difference and we audit them later.

Also ignore {{Code|(mount_folder) /usr/share/derivative-maker_timestamp}} for now, this is an expected difference and we audit them later.

Also ignore {{Code|(mount_folder) /var/lib/initramfs-tools/...}} for now, this is an expected difference and we audit them later.

There should be no other differences in the {{Code2|.report}} files!

Now it is time to audit the differencing files which we ignored above.

Diff the virtual machine description file {{Code|(extracted_ova_folder) *.ovf}}. Only the uuids should differ. (The extracted_ova_folder folder can be found in your report_temp folders.)

Diff the manifest file {{Code|(extracted_ova_folder) *.mf}}. Only the checksums should differ.

Finally diff the rest of the files in your {{Code2|manual_analysis_folder}}. Using a diff application which can diff whole folders such as {{Code2|meld}} would be useful, but what tools you use is up to you. Have a look at the "Expected Differences" chapter above for the kind of differences you can expect.

Auditing these files is unfortunately a bit difficult. The best tip to be given is to use your common sense. For example, having two {{Code2|/etc/init.d/.depend.start}} files at same size, but different order is probably not a backdoor. But if there are a lot extra lines looking weird ([https://security.stackexchange.com/questions/40072/could-someone-explain-parts-of-the-fbis-firefox-0-day/40108#40108 unrelated example code]), there might be something wrong.

== Verifiable {{project_name_short}} Debian Packages ==
{{Verifiable_Pkgs_Introduction}}

Shouldn't they match, you could also compare them using {{Code|debdiff}} or {{Code2|debbindiff}}.

We don't have a script yet to automate that process.

== Implementation ==
In case you like to review the implementation, the related scripts can be found here:

* Remove most non-deterministic parts (auto generated files which always look at bit different): https://github.com/Kicksecure/initializer-dist/blob/master/usr/libexec/initializer-dist/chroot-scripts-post.d/80_cleanup
* Build step to create the report: https://github.com/{{project_name_short}}/derivative-maker/blob/master/build-steps.d/5100_create-report
* The analyze image script, which creates the report: https://github.com/adrelanos/{{project_name_short}}/blob/master/help-steps/analyze_image
* The user interface locally re-creating non-deterministic files when {{project_name_short}} starts for the first time:
* Re-creating non-deterministic files when {{project_name_short}} starts for the first time: todo

= Discussion =
* [https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20131209/000009.html Announcing {{project_name_short}} First Implementation of Verifiable Builds]

= Links =
* https://reproducible-builds.org/contribute/debian/#Working_on_installation_media_or_live_systems
* [https://phabricator.whonix.org/maniphest/?statuses=open%2Creview&allProjects=PHID-PROJ-6h24kenaoc2givyt6uho#R {{project_name_short}} Verifiable Builds Issue Tracker]

= Footnotes / References =
[ {{project_name_short}} 8 snapshot of this page]

{{reflist|close=1}}

{{Footer}}

[[Category:Documentation]]
[[Category:Development]]