Netatalk 4.0 - Future-proofing Apple File Sharing

The current Netatalk release series (2.x and 3.x) are finally in a good place where we have a modern and flexible build system, with fully functioning user authentication. I can finally pivot to what I wanted to do in the first place: consolidate the best of 2.x and 3.x into one Mac file sharing suite to rule them all.

I hereby present: The Netatalk 4.0 Roadmap.

The mission statement is: A future-proof file sharing suite for Apple //e to macOS and everything in-between.

In the long run, I want to stop supporting two very different branches and have one branch will all the code modernization of 3.x with the support for old clients that 2.x provides.

At a high level, these are the projects:

AppleTalk

- Graft the AppleTalk protocol modules back onto libatalk: asp, atp, nbp, ddp
- Bring back the AppleTalk daemons: atalkd, papd, timelord, a2boot
- Bring back the AppleTalk networking tools: aecho, getzones, nbp, pap
- Bring back afpd support for AFP 1.1, 2.0, and 2.1

Modern Tech

- Write a CNID backend in SQLite (Berkeley DB is abandonware)
- Write a Spotlight indexing and search backend in Elasticsearch (Gnome Tracker has not been working well)
- Use Nettle as crypto backend (eliminate the remaining dependencies on OpenSSL)
- Use GDBus as D-Bus client (dbus-glib is going away)

On top of this, I want to remove the old Autotools build system, shore up all the insecure memory mangement, write unit tests...

Needless to say, this will be a massive undertaking. I am looking for collaborators to make this happen. If you or someone you know has a knack for C (or is willing to learn) I can offer a job that is unpaid but comes with the glory of keeping the dream of Apple file sharing alive for the next generation. :)

All of the relevant code from 2.x has now been restored to the main development branch, and the build system has been retooled to build it.

The next step will be to adapt the various interfaces that were refactored or simplified during the 3.0 release cycle. Notably, the daemon child process handling was simplified to only support DSI (TCP) processes, so it needs to be reworked to manage ASP (AppleTalk) AFP processes again.

If you want to keep track of the progress (and perhaps contribute in some form) see the "appletalk" label in the issue tracker.

We are now looking for alpha testers of Netatalk 4.0.0.

It's a one-way migration path from Netatalk 2.x, so please upgrade only on AFP volumes that have backups, or create fresh volumes for testing.

4.0.0 has a handful of new dependencies, not related to AppleTalk. Notably, Nettle and the Unicode DB. See this pre-release build appendix for concrete packages and commands: https://netatalk.io/4.0/htmldocs/compile ... To get started, clone the latest "main" branch in the Netatalk git repo, and build it with Meson!

Netatalk has to be built with the "-Dwith-appletalk=true" and then you have to put "appletalk = yes" in afp.conf to enable the transport layer. As with 2.x, your OS has to supply appletalk (DDP) networking in the kernel for this to work.

What we want to ascertain right now is twofold: Does AFP-over-AppleTalk work in your environment and workflow. And, is there no performance degradation (file listing, transfer, deletion, etc.)

Looking forward to hearing your feedback!

One thing I haven't extensively tested is the conversion of Apple Double data from the old .AppleDouble folder format to native extended attributes and "dot" files (ex: ._MyFile). So..... back up your shares before trying this! The code should work fine as it wasn't really touched, but always use caution regardless. So far I've tested the following clients:

AppleShare Workstation 1.1 (AFP1.1)
Apple IIgs System 6.0.1 (AFP2.0)
Apple IIgs System 6.0.1 using AFPBridge to connect via TCP/IP (AFP2.0)
The ProDOS 8 2.0.3 client bundled with the above (AFP2.0)
System 7.1 with the stock ASC 3.5 (AFP2.1)
MacOS 8.1 with the stock ASC 3.7.4 via both AppleTalk and TCP/IP (AFP2.2)
macOS 10.15 via TCP/IP (AFP3.4)

Eventually I'll get an XP/2003 image spun up and try Services For Macintosh and PCMacLAN.

Most of this was via emulated machines, so real hardware testing is always appreciated. Get out your rare clients like an Apple IIe Workstation card, or AppleShare for MS-DOS and break this thing! Modern TCP/IP client testing is appreciated as well as I'm concerned about any possible regressions.

Netatalk 4.0.0 beta is available as a Docker image now. It comes with all the best parts of the 3.x release series, and the 2.x release series that you all know and love!

https://hub.docker.com/r/netatalk/netatalk

If you have a host computer with an AppleTalk network stack (Linux or NetBSD) you only need to follow the instructions to get started. All options are named in the same way as the previous Docker releases.

4.0.0 is out now!

[Image: github.com]

Release Netatalk 4.0.0 - Netatalk/netatalk

Netatalk 4.0.0 is available! The Netatalk team is proud to announce the first release in the Netatalk 4.0 release series. This release constitutes major changes over Netatalk 3.2, and is recommende...

[Image: github.com] github.com

If you are a brave early adopter, why don't you take it for a spin today? :)

>> rdmark said:

4.0.0 is out now! Release Netatalk 4.0.0 - Netatalk/netatalk Netatalk 4.0.0 is available! The Netatalk team is proud to announce the first release in the Netatalk 4.0 release series. This release constitutes major changes over Netatalk 3.2, and is recommende... github.com If you are a brave early adopter, why don't you take it for a spin today? Click to expand...

Assembling packages has always been fascinating for me.

The first attempt to run "meson setup build" failed:

meson.build:544:4: ERROR: Problem encountered: UnicodeData.txt not found. Specify path with -Dwith-unicode-data-path

I decided it like this

curl -O https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt

The second attempt to run "meson setup build" failed:

meson.build:1555:4: ERROR: Problem encountered: xsltproc and docbook-xsl stylesheets are required to compile manpages and html documentation

And this puzzles me, because the manual says about the optional requirement of xsltproc

[manual: Optional: avahi or mDNSresponder (for Zeroconf support) cracklib and cracklib dictionary (for password strength check in afppasswd) Docbook XSL and xsltproc (for manpages & manual documentation) GLib 2 and D-Bus (for afpstats support) Kerberos V (for krbV UAM support) libacl (for ACL support) libldap (for LDAP support) libpam (for PAM support) libtirpc or libquota (for Quota support) tcpwrap (for TCP wrapper support)]

Moreover, brew does not know such a package.

netatalk % brew search xsltproc                  
Error: No formulae or casks found for "xsltproc".

Maybe there is some way to ignore it?

@V.Yakob This page has some hints how to build on macOS with Homebrew: https://netatalk.io/stable/htmldocs/compile#build-macos

In short, the xsltproc tool is in the `libxslt` bottle.

@rdmark На первый взгляд, всё хорошо.
Now this PM8100 copies several large files for testing.

[Versions: /opt/homebrew/sbin/netatalk -V netatalk 4.0.0 - Netatalk AFP server service controller daemon /opt/homebrew/sbin/afpd -V afpd 4.0.0 - Apple Filing Protocol (AFP) daemon of Netatalk]

At first glance, everything is fine. (y)

In the last line of the installation output, I noticed that the old approach of installing daemons is used

Running custom install script '/bin/launchctl load -w /Library/LaunchDaemons/io.netatalk.daemon.plist'

Maybe it's better this way?

launchctl bootstrap system/ /Library/LaunchDaemons/io.netatalk.daemon.plist
launchctl kickstart system/io.netatalk.daemon

@V.Yakob Wonderful, thanks for sharing!

Interesting, I didn't know launchd daemon management procedure had changed. Do you have a GitHub account? The best way to track this would be if you could file a bug over at the issue tracker. We can confirm the details and fix it in a future release.

[Image: github.com]

Build software better, together

GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.

[Image: github.com] github.com

@rdmark, Changes have been there for a long time, starting with Yosemite. There is a good cheat sheet for managing demons and services, and even a graphic application (LaunchControl) for symbolic money.
Yes, I have an account on GitHub, I'll write.

I only noticed this thread today. I'm big on "layman's terms" and dummying down difficult concepts into a form digestible by mortal man.
You know...
FOR THE REST OF US.

I Googled a bit to figure out even what Netatalk is, but that led me to a bunch of techno jargon that made my eyes glaze over. Although my experience with ChatGPT4o has been mixed, I decided to give it to summarize and simplify, so my feeble brain can better understand Netatalk. Specifically, I asked ChatGPT4o to give me a proper explanation in simple terms, and then asked it to tell me how Netatalk 4.0 is installed, and then I asked why I would even need it versus just using a FloppyEMU and BlueSCSI like I do now. Below is what it told me. Feel free to correct the info below appropriate.

The main reason I feel this kind of explanation important is because when people understand what a given thing is, that understanding will often drive more interest in it. This appears to be an excellent project that could use more participants.

Netatalk Overview

Netatalk is a piece of software that allows Unix-like operating systems (like Linux or BSD) to act as file servers for Macintosh computers. Essentially, it helps different types of computers talk to each other and share files easily.

Here's why you might find Netatalk useful:

File Sharing Across Different Macs: Since you have both modern and vintage Macs, Netatalk can help you share files between them seamlessly. For example, you can use a Unix-based server to store files that can be accessed by both your old Macintosh IIci and your new MacBook Pro.
Compatibility with Older Systems: Netatalk supports the Apple Filing Protocol (AFP), which is particularly useful for older Macs that might not support newer file-sharing protocols like SMB (used by modern macOS versions). This means you can still use your vintage Macs to access and store files on a modern server.
Centralized Storage: If you set up a Unix-based server with Netatalk, you can centralize your file storage. This makes it easier to manage backups and ensures that all your Macs, regardless of their age, can access the same files.
Time Machine Backups: Netatalk can also be configured to support Time Machine backups, allowing you to back up your modern Macs to a Unix-based server.

In summary, Netatalk is like a bridge that connects your modern and vintage Macs, making it easier to share files and manage backups across different systems.

General Installation and Setup Netatalk 4.0

Installing Netatalk 4.0 can vary in complexity depending on your familiarity with Unix-like systems. Here's a simplified overview:

Installation: You typically need to install it via your system's package manager (like apt for Debian-based systems or yum for Red Hat-based systems). For example, on a Debian-based system, you might use:
sudo apt-get update sudo apt-get install netatalk
Configuration: After installation, you'll need to configure the afpd (Apple Filing Protocol daemon) by editing the afp.conf file, usually located in /etc/netatalk/. This involves specifying the directories you want to share and setting up user permissions.
Starting the Service: Finally, you start the Netatalk service with:
sudo systemctl start netatalk sudo systemctl enable netatalk

While these steps are straightforward for someone with Unix experience, they might be a bit challenging if you're new to it. There are plenty of online guides and communities that can help you through the process.

Installing Netatalk 4.0 on macOS:

Install Homebrew: Homebrew is a package manager for macOS that simplifies the installation of software.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Install Dependencies: You might need additional libraries and tools. For example:
brew install autoconf automake libtool

Download and Compile Netatalk: You would need to download the Netatalk source code and compile it.

git clone https://github.com/Netatalk/Netatalk.git
cd Netatalk
./bootstrap
./configure
make
sudo make install

Configure Netatalk: Edit the configuration files to set up your shares and permissions, similar to how you would on a Unix-like system.

Considerations:

Complexity: This process can be quite complex and may require troubleshooting.
Support: There is limited documentation and community support for running Netatalk on macOS compared to Linux or BSD.

Alternative:

If you prefer a simpler setup, using a Linux-based virtual machine (VM) on your Mac to run Netatalk might be a more straightforward solution. This way, you can leverage the full support and documentation available for Netatalk on Linux while still using your Mac.

Advantages of a Netatalk Server vs. Using SD Cards with FloppyEMU or BlueSCSI

Netatalk Server Advantages:

Centralized Access: All your vintage and modern Macs can access the same files stored on the server without needing to physically transfer SD cards.
Ease of Use: No need to constantly swap SD cards between devices. You can access files directly over the network.
Backup and Redundancy: Easier to set up automated backups and ensure data redundancy on a server.
Time Machine Support: You can use the server for Time Machine backups for your modern Macs.
Scalability: As your collection of vintage Macs grows, you won't need additional physical storage solutions.

Using SD Cards with FloppyEMU or BlueSCSI:

Simplicity: No need to set up and maintain a server. Just copy files to an SD card and use them.
Portability: Easy to move files between different vintage Macs without network dependency.
Cost: No need for additional hardware or a dedicated server.

In summary, a Netatalk server offers more convenience and advanced features, especially for managing a large number of files and devices. However, using SD cards with FloppyEMU or BlueSCSI is simpler and more portable, which might be sufficient for occasional file transfers.

@JDW You are absolutely right, I've been neck-deep in this project for three years now so I sometimes forget that not everyone knows everything about it. :)

The AI did a pretty good job at summarizing the pros and cons. I might steal some of those blurbs for the homepage or wiki landing page. Unsurprisingly, its data is a bit old. It has outdated information about how to build Netatalk.

For macOS, this is the latest for Netatalk 4.0.0: https://netatalk.io/stable/htmldocs/compile#build-macos

We also have this wiki page, which has a mix of new and old information (as it goes with a wiki): https://github.com/Netatalk/netatalk/wiki/Installing-Netatalk-on-macOS

FWIW, I've gradually been trying to make f.e. the top section of the homepage a little bit more friendly to first-time users: https://netatalk.io/

I'd love to hear concrete suggestions on how to better promote and explain Netatalk!

>> rdmark said:

I'd love to hear concrete suggestions on how to better promote and explain Netatalk! Click to expand...

I think a large number of vintage Mac enthusiasts are always looking at what's new in the community and examining what they may not have heard of before. Many of these people approach a new thing with this fundamental question: "What can this thing do for me?"

The first sentence on your website says this:

Netatalk is a Free and Open Source file server that implements the Apple Filing Protocol (AFP).

After reading that, someone totally oblivious to the advantages of "file servers" or "AFP" may wonder: "I currently use a Blue SCSI or FloppyEMU to move files between my modern Mac and my old Macs. Is Netatalk something I really need? Can it benefit me in a way I can understand and easily use? Or is it just for people smarter than me in some kind of specialty use case?"

If common questions like that can be answered in super easy layman's terms, I think Netatalk then can be better understood and possibly more desirable to have by a larger number of people. In other words, it's not about telling a newbie "here's how our file server works." It's about explaining: "here's why you could stand to benefit from this."

Why do I want AFP? Why don't I have AFP already? When did I stop having AFP?
I have Intel Macs (Mac OS X10.4 to 15.0) and PCs (Linux/Windows).
I have PowerPC Macs that can run System 7 to OS X 10.5. They can connect to Intel Macs that are running early versions of Mac OS X (either booted or in a VM). It would be beneficial to not have to reboot to an older OS or start a VM.

I think I remember installing netatalk on a jail broken iPod touch and/or iPad in the past. I could then use it to transfer files to the iPod or iPad for emulators or whatever.

@JDW I can categorize the Netatalk "customers" in three rough categories:

1. Enterprise. Specifically, IT or SRE folks that operate large scale deployments at offices or schools. This category is probably shrinking, but they still exist and are pretty vocal.

2. Regular home users. Folks using Netatalk primarily with modern Macs. Time Machine backups seems to be a popular use case. Long-time users swear by Netatalk because transfer speeds tend to be better than Samba/SMB.

3. Retro Mac users. Y'all. :)

The most effective communication strategy should strike a balance between the expectations and needs of all user segments. I don't want to position Netatalk as a retro Mac solution, exclusively.

>> rdmark said:

The most effective communication strategy should strike a balance between the expectations and needs of all user segments. I don't want to position Netatalk as a retro Mac solution, exclusively. Click to expand...

Understood. And for the record, I myself fall into your categories (2) and (3) because I have Macs ranging from 1984 to present day, and I use Time Machine for backups of my modern Macs.

>> V.Yakob said:

@rdmark , Done . Click to expand...

Thank you! I left a two questions for you in the issue tracker. Please have a look when you got the opportunity.

>> joevt said:

Why do I want AFP? Why don't I have AFP already? When did I stop having AFP? I have Intel Macs (Mac OS X10.4 to 15.0) and PCs (Linux/Windows). I have PowerPC Macs that can run System 7 to OS X 10.5. They can connect to Intel Macs that are running early versions of Mac OS X (either booted or in a VM). It would be beneficial to not have to reboot to an older OS or start a VM. I think I remember installing netatalk on a jail broken iPod touch and/or iPad in the past. I could then use it to transfer files to the iPod or iPad for emulators or whatever. Click to expand...

You make a good point. If you have a fleet of old Macs you can definitely run their native AFP file servers, but using such old machines as permanent servers is not advisable, IMHO. In addition to the inconvenience: Old computers are energy inefficient, run insecure software, and you risk wearing down the aging electronics at the same time.

The old iPhone port of Netatalk was a super cool hack indeed. The era of easy-to-jailbreak iOS devices is in the distant past unfortunately. In this day and age I think a Raspberry Pi is a great option for an energy efficient and compact file server!