View source for Building Gramps AIO cx freeze-based

{{man warn|Gramps 4.2.8 and later.|Any comments and help (e.g. on wiki pages) are welcome.}}

The '''Gramps''' "software bundle" for Windows or '''GrampsAIO''' is an all-in-one (a.k.a. '''"AIO"''') installation package of Gramps which includes all dependencies for the Windows platform.

This tutorial describes how to generate a Windows AIO package from a development installation such as is created with [[Gramps for Windows with MSYS2]].

It is intended for Windows users and does not attempt to describe how to do the same thing on Linux or a Mac.

("AIO" means all-in-one, which means that everything you need to run Gramps is included in the one AIO file you download and then install -- all the related programs and library files which Gramps needs on Windows.)

{{man note|Note|The first Gramps AIO package was created by an advanced Windows power-user named Josip.  He is the one who evolved the AIO package and modified the AIO package over the years as Gramps has changed.  Everything documented here initially done by him; this just documents some changes and tries to explain it for others.  Any Gramps Windows user owes him a large debt of gratitude.}}

{{man tip|Note|In this tutorial you will be creating a 32-bit AIO.  The steps for creating a 64-bit version are quite similar, just a few editing changes to the files and scripts. }}

==Prerequisites==
You must have a development version of Gramps up and running.  See [[Gramps for Windows with MSYS2]] which describes the process for a 64-bit installation.  When following this process, you must change a few things for 32-bits.

{{man note|Note|Pay close attention to the shell to use when running the various commands.  Use <code>MSYS2 MSYS</code> for the <code>pacman</code> and <code>makepkg-mingw</code> steps and the <code>MSYS2 MINGW32/64</code> shell to run <code>python3 Gramps.py</code>.}}

When the instructions say to use the <code>MSYS2 MINGW64</code> shell you should use <code>MSYS2 MINGW32</code> shell instead for 32-bit operation.

Change the 'pacman' command install file prefixes from <code>mingw-w64-x86_64</code> to <code>mingw-w64-i686</code>.

Because Josip does not have (as of this writing) some of the packages (gtk, db, bsddb3) that he built done for 32-bit we have to do them ourselves 'the hard way'.

Change the <code>MINGW_INSTALLS=mingw64 makepkg-mingw –sLf</code> lines to <code>MINGW_INSTALLS=mingw32 makepkg-mingw -sLf</code>.

=== Get the tools ===

{{man note|Note|Josip uses a cx_Freeze with some changes he made to make the process a bit easier.  I attempted to make similar changes, but we will have to build the utility. }}
Download https://gramps-project.org/wiki/images/a/a3/Mingw-w64-python-cx_Freeze.zip [[:File:Mingw-w64-python-cx_Freeze.zip| (stored here)]] to a convenient location with your browser.

I do my builds in the <code>c:\msys64\home\user\builds</code> directory, so we will make a new sub-directory for cx_Freeze there.

I use 7-zip to extract to the <code>c:\msys64\home\user\builds\cx_Freeze</code> directory.


{{man note|Note|Use <code>MSYS2 MSYS</code> shell.}}
  MINGW_INSTALLS=mingw32 makepkg-mingw -sLf
  pacman -U mingw-w64-i686-python3-cx_Freeze-5.0.2-1-any.pkg.tar.xz

The patches included in the zip should fix up cx_Freeze for a bug where it tries to load an sqlite3.dll file, and to add properly optimized base files for Python startup (In the original version the 'optimize' functionality doesn't work).


NSIS is a tool to build the actual installer.  As of NSIS version 3.05 the standard nsis available from the MINGW archive works.
  pacman -S mingw-w64-i686-nsis

If you have NSIS version 3.04, The version in the mingw system has a bug that makes displaying of the font files during the AIO install messed up.  I've created a patch, so we need to build our own.
Download https://gramps-project.org/wiki/index.php/File:Mingw-w64-nsis.zip [[:File:Mingw-w64-nsis.zip| (stored here)]] to a convenient location with your browser.

Unzip and cd into the directory then:
{{man note|Note|Use <code>MSYS2 MSYS</code> shell.}}
  MINGW_INSTALLS=mingw32 makepkg-mingw -sLf
  pacman -U mingw-w64-i686-nsis-3.0.4-1-any.pkg.tar.xz

=== Get a reference AIO ===
We also need the most recent version of Gramps AIO available.  If it is for your branch, it is likely to have the closest scripts with minimal editing.  Of course a newer version for a newer branch might have some changes in the scripts that deal with more recent dependencies.  You may want to reference both in updating scripts for the new version.  Get it installed so you can see the files from that installation.

=== Install the old Gramps ===
When you start the installer, it will ask you to select language and spell check files.  Select all of them.
It will then ask where you want it installed.  You can install it where you want, as it is only a temporary location which you may delete afterward.  But make a note of the folder since you will need to know it later.  Also, it needs to be a new (empty) folder, not an already existing one.

In this tutorial we'll say you tell the installer to put it into a new "<code>C:\Program Files (x86)\GrampsAIO32-5.1.0</code>" folder.

Then double-click on the AIO executable to start the 5.1.0 installation.  (If your Windows asks whether you will allow that program to change things on your computer, say yes of course.)

You can put it anywhere you want but if you do, change the string "<code>C:\Program Files (x86)\GrampsAIO32-5.1.0</code>" in this tutorial to the actual name you used.  But every time I use "<code>C:\Program Files (x86)\GrampsAIO32-5.1.0</code>" I mean the top of the tree where you installed the "old" AIO.

After the installer is finished, the last window will offer you the opportunity to "Run" the Gramps you just installed, so you should un-click that box before you hit "Finish", since there is no need to run the old one.

Similarly, you can delete the "GrampsAIO32 x.x.x" (or GrampsAIO64 one)" icon shortcut from your Desktop, since you won't be using it.

=== Get Gramps installed in our development environment ===
{{man note|Note|Use <code>MSYS2 MINGW32</code> shell.}}

If you have not already done so as part of the prior steps, do the following:
  cd ~/grampsdev  # my copy of git repo checked out with the branch you are building.

  python3 setup.py build

To install preparatory to creating AIO do 
  python3 setup.py install

== Building the AIO ==
Note: Gramps contains some spell checking files that are NOT installed by the previous 'pacman' installs.  I found it easiest to copy them from the existing Gramps installation (where I installed all the language and spelling files during the AIO install).
The originals were in the installation <code>/share/enchant/myspell</code> directory.
I copied the whole 'myspell' directory above to my <code>c:\msys64\mingw32\share\enchant</code> directory so they could be found by the next step scripts.  I used the Windows file explorer as that is easiest for me, but if you are familiar with the Linux <code>cp</code> command, you can do it from the MINGW32 shell.

Before doing the AIO build, you may want to compare the existing <code>src/setup.py</code> and <code>src/grampsaio32.nsi</code> scripts with the latest ones available from other AIO installations.  They change depending on the contents of the AIO and may point up other dependencies or changes.  I use Notepad++ and its 'Compare' plugin for this, but feel free to use whatever tool you like.

I use dir <code>~/aio</code> where my <code>setup.py</code>, <code>grampsX.py</code>, icons and nsis script is stored.

  cd ~
  cp -r "/c/Program Files (x86)/GrampsAIO32-5.1.0/src" aio

=== Modifying the setup.py script ===
The setup.py script should be located in <code>~/aio/setup.py</code>.  Use you favorite editor to make sure following is right:
* Python versions are correct; At the moment, Python3.6 is in use, however if your version is different, adjust the references to <code>python3.6</code> and <code>cpython-36m</code> and <code>python36</code> appropriately.
* The 32/64 bit differences are correct; <code>gspawn-win64-helper</code> may need to change to <code>gspawn-win32-helper</code> or vice versa.
* If you have added new dependencies, find the right place to put them into the script.

Then use the following to create <code>~/aio/mingw32</code> with all files needed for bundle.
=== Modifying the gramps.nsi script ===
Open the <code>gramps.nsi</code> file with your favorite editor.

Part of the contents should look something like the following:
<pre>
!define APPNAME "GrampsAIO32"
!define APPVERSION "5.1.0"
!define APPBUILD "1"
!define APPNAMEANDVERSION "${APPNAME} ${APPVERSION}"
!define APP_PUBLISHER "The Gramps project"
!define APP_WEB_SITE "http://gramps-project.org"
BrandingText "Version ${APPVERSION}-${APPBUILD} -- a new maintenance release"
</pre>

{{man note|Make sure your editor works|One user reported unicode problems with Notepad and then downloaded [https://notepad-plus-plus.org/ Notepad++].}}

===== Increase the version number =====
Change the '''APPVERSION''' line from "5.1.0" to "5.1.1" instead (or higher if making a newer version).

===== If needed update APPNAME and APPBUILD =====
Possibly APPNAME and APPBUILD will need changing too, depending on what you are trying to do.  Generally the APPBUILD gets incremented if a second AIO release is made from the same version (because of a building issue.

{{man note|If you are a power user|and have downloaded something other than a tar.gz file, see [[GrampsAIO-4_package_updating#Appendix_A|Appendix A]] to change APPBUILD.}}

If necessary, change the BrandingText line to have the name of the new Gramps release instead of the old release name. Usually only necessary for major version changes.

Then save that gramps.nsi file.

=== Run the scripts ===
  python3 setup.py build_exe --no-compress

Then I pass nsis script from ~/aio/mingw32/src to nsis.

  makensis ~/aio/mingw32/src/grampsaio32.nsi

Note that after the compilation, the resulting files will be compressed.

{{man warn|Please be patient|The last part of the setup.py script and the NSIS package compression will probably take a while<br>'''-- OVER FIVE MINUTES FOR ME --'''}}

{{man note|Experimenting with NSIS compression|See [[Building Gramps AIO#Appendix_B|Appendix B]] for power-user details, at this stage.}}

The result is in <code>C:\msys64\home\user\aio\mingw32\src\GrampsAIO-5.1.1-1_win32.exe</code>.

{{man warn|If you do this more than once, make sure to delete the result file, or it will get encoded into the next AIO. }}

== Debugging ==
I had several issues in this process.  First of all, the mingw repositories are "rolling release", so every time you do a <code>pacman –Syuu</code> you will have different packages and potentially new issues.

Take care to make sure that you don't overwrite any of the 'static' Gramps dependencies (gtk 3.18.9, bsddb 6.1.0, db 6.0.30 at the time of this writing) with newer versions, and if you do, use <code>pacman -U file.tar.xz</code> to revert them.  Get this right FIRST, because if you build anything based on one of the static dependencies, it may end up not working on a newer version.  If you do build something out of order, you may end up with mysterious import failures with messages that don't tell you what is really wrong.  And it's really hard to debug the installed AIO, as it doesn't look like a typical Python process to the usual debuggers.  Experience speaking.

You may have to build one or more dependencies "the hard way"; that is download the PKGBUILD and any patches from a suitable source <code>https://github.com/Alexpux/MINGW-packages/trunk/mingw-w64-osmgpsmap-git</code> for example, and fix it up for newer versions and build it.  See <code>https://github.com/msys2/msys2/wiki/Creating-packages</code> for help here.  Or even download source code and compile yourself.

I used the old DOS <code>tree /F /A > aionew.txt</code> command to make filename dumps from both the new AIO and a recent older one that was freshly installed in an empty directory.  This ensures that there are no '.pyc' files to confuse the issue.  Then I did a difference compare (using Notepad++) of the two to see what files are different.  Any missing files in your new AIO are suspect (unless you did this deliberately).


=== Appendix A ===
==== Indicating newer APPBUILD's ====
If you are a power user, you might have downloaded or used git to checkout the latest code in the grampsxx branch:

  https://codeload.github.com/gramps-project/gramps/zip/maintenance/gramps51

but in any case, if you have downloaded something other than a gramps-release tar.gz (or .zip) file, say this in your top directory <code>~/grampsdev</code> (assuming you have installed Git for Windows):

  git rev-list --count HEAD

and then

  git rev-parse --short HEAD

and then put an "r" in front of the first string, followed by a hyphen ("-") and the second string, and then use the resulting combined string as your APPBUILD in the <code>gramps.nsi</code> file, for clarity.

So for example your APPBUILD line might look like this:

  !define APPBUILD "r23381-25018d6"

Then continue with the [[GrampsAIO-4_package_updating#Increase_the_version_number|NSIS package building]]

==== Developers uploading to Sourceforge ====

If you are a developer and plan to upload it to SourceForge and it is an AIO of (a forthcoming) "master" put it into:

  http://sourceforge.net/projects/gramps/files/Unstable

but if it is an AIO for a maintenance branch, make a Snapshots folder in that branch and put it there, e.g.:

  http://sourceforge.net/projects/gramps/files/Stable/5.1.1/Snapshots

so then the APPBUILD numbers will uniquely define that AIO.

But note that the NSIS-generated filename will say (for instance) "5.1.2" even though it is really a "5.1.1" with bugs fixed, which is why it belongs in 5.1.1/Snapshots.

Also see [[What to do for a release]]

=== Appendix B ===
====Experimenting with NSIS compression====
If you are a power user and are experimenting with slight changes to the gramps.nsi script, you can drastically shorten the time needed to have NSIS generate an AIO package by commenting out these two lines (with a semicolon, a ";"):

  SetCompressor /SOLID /FINAL LZMA
  SetCompressorDictSize 128

But if you are building an AIO which will be released to the public (put onto the official gramps files repo), you need to be '''very sure''' to uncomment '''both''' those lines '''first'''.

==See also==
* [[All In One Gramps Software Bundle for Windows]] for the Gramps 3.4.x and earlier series
* [[Gramps for Windows with MSYS2]]
* [[GrampsAIO cx freeze-based]]

[[Category:Documentation]]
[[Category:Developers/General]]
[[Category:GEPS|W]]
[[Category:Developers/Packaging]]