In this post, I’ll be describing the parts of the workflow I use when working on compiler features. If you work on other parts of Swift (say the Swift standard library), or on other aspects of the compiler (like performance), this might not be helpful.

This post assumes that:

• The Swift source code repositories are checked out under ~/swift-source/
• The Swift repository is at ~/swift-source/swift
• The Swift repository has two remotes, origin and fork: origin points to Apple’s Swift repository on GitHub, and fork points to our fork of that repository on GitHib
• The SWIFT_BUILD_DIR environment variable is set to a Swift build directory

My previous post talks about one way to get a setup like that.

My workflow uses the following parts. The order of using these parts varies depending on what I work on.

• Starting off
• Debugging
• Making changes
• Syncing up with master
• Testing
  • Run the testsuite
  • Add to the testsuite
  • More on the testsuite
• Open a pull request
• Modify a pull request

Starting off

When starting to work on a fix or feature, we should create a new branch off master:

$ git checkout -b <feature-branch> master

Debugging

To debug a compilation of swiftc file.swift, we can’t just run lldb -- swiftc file.swift because swiftc invokes itself and other executables as separate processes to do the actual compilation.

To see the underlying commands without running them, we can run:

$ swiftc file.swift -###

Typically, we want to run the debugger on the first command printed, so we can say:

$ lldb -- `swiftc file.swift -### | head -n 1`

The DebuggingTheCompiler.rst document in the Swift repository has some helpful information on debugging. I’m keeping some notes as well.

Making changes

When committing, we should follow the Swift commit message guidelines.

To rebuild Swift after making changes, we can run:

$ cd $SWIFT_BUILD_DIR
$ ninja swift

If there were no changes to header files, rebuilding Swift would typically take only a few minutes.

Syncing up with master

If we’ve been working on our feature branch for a while (say a week), and if we have commits as work-in-progress on our feature branch, it would be a good idea to rebase our feature branch on top of the latest commits in master.

Assuming we’re on the feature branch:

$ git fetch origin master
$ git rebase origin/master

If the are conflicts, they would have to be resolved during the rebase.

After rebasing to master, we’ll have to rebuid Swift again.

$ cd $SWIFT_BUILD_DIR
$ ninja swift

We’re rebuilding only Swift, while the changes we fetched on master could have included changes to other parts of the project, like llvm and the Swift standard library.

• In case the previously built version of Swift standard library is out of date, we’ll get an error like this when running swift or swiftc:

  error: compiled module was created by an older version of the compiler; rebuild 'Swift' and try again
  

  Then we should rebuild the standard library as well, like:

  $ cd $SWIFT_BUILD_DIR
  $ ninja swift swift-stdlib
  

• In case the previously built version of llvm is out of date, we will get errors referring to llvm header files while compiling Swift. In that case, we should rerun the build script.

  First, we update all the repositories:

  $ cd ~/swift-source/swift
  $ git checkout master
  $ cd ~/swift-source
  $ ./swift/utils/update-checkout
  

  If there are commits in our feature branch, we can switch to that branch now:

  $ cd ~/swift-source/swift
  $ git checkout <feature-branch> # rebase again onto master if reqd.
  

  We can now run build-script with the same options as the initial build. Incremental runs of build-script take lesser time than clean runs.

Testing

Run the testsuite

To make sure our changes haven’t affected working parts of the compiler, we should test our version of Swift against the testsuite. The expected output is embedded inside the test file, and the output is verified using llvm’s FileCheck utility.

We use the lit.py script for that. The following commands assume that lit.py is available in our PATH.

By default, lit.py prints one of “PASS:”, “FAIL:” or “XFAIL:” (which means expected failure) for every testcase. Passing -s suppresses that.

If we pass -v, lit.py shall print the commands it invoked for failing tests, so we can try to run it ourselves.

Typically, we know our changes are likely to affect only certain aspects of the compiler, so we run only certain tests:

• To run tests under a certain test directory:

  $ lit.py -sv $SWIFT_BUILD_DIR/test-linux-x86_64/<test-directory>
  

  where <test-directory> is a directory path under ~/swift-source/swift/test/.

• To run tests matching a filename pattern:

  $ lit.py -sv $SWIFT_BUILD_DIR/test-linux-x86_64/<test-directory> --filter=<pattern>
  

If we’re about to open a pull request, we might want to run the full testsuite:

$ lit.py -s $SWIFT_BUILD_DIR/test-linux-x86_64/

Add to the testsuite

Most changes to the compiler will require that the change be tested. In that case, the pull request should include an addition to the testsuite to verify the change.

More on the testsuite

• Testing.md in the Swift repository
• codafi’s post on the Swift forums

Open a pull request

To open a pull request, we should:

1. Push the feature branch to our fork

   $ git push fork <feature-branch>
   

2. Go to your forked repository on GitHub on a web browser, where you should see an option to create a pull request

3. When opening the pull request, we should:

   • include the bug number resolved by the pull request
   • @-mention a potential reviewer, ideally from the Swift team

   If it’s not obvious who should review something, we could look at CODE_OWNERS.TXT in the Swift repository.

Once we open a pull request, we should wait till a reviewer takes this up for review. Only those who have commit access can ask the @swift-ci build bot to invoke automated testing on the pull request.

Modify a pull request

We might have to change a pull request for different reasons:

• to make corrections in the implementation
• to take a different approach
• to address conflicts with changes in the master branch

If we add commits to the same branch and push that to the same branch on our fork, and they will be added to the pull request automatically.

$ git push fork <feature-branch>

If we end up with a different set of commits altogether (maybe we used interactive rebase to edit the commit history, or we rebased to a different base commit), we can force-push to the same branch on our fork, and the pull request will be updated automatically.

$ git push -f fork <feature-branch>

Force-pushing can sometimes cause the @swift-ci build bot to report an error even when there’s no error. In case the build bot reports an error, we should take a look at its logs to check if it’s a genuine error or not.

If you have questions, corrections, or feedback on this workflow, please let me know.

If you’re interested in starting on Swift compiler development, renting out a remote server like this can be a great way to get your feet wet.

So, let’s get started on setting up a remote server for Swift compiler development.

• Choose a remote Linux machine
• Set up the remote Linux machine
  • Set up a user
  • Set up SSH keys
  • Set up tmux
• Set up the local macOS machine
  • Set up iTerm2
  • Using iTerm2’s tmux integration
  • Configuring iTerm2’s tmux integration
• Build Swift
  • Get the dependencies
  • Get the code
  • Build it
• Set up the development environment
  • Git
  • Vim
  • lldb
• Set up our fork of Swift

Choose a remote Linux machine

Choosing a machine configuration depends on the system requirements for building Swift.

For hacking on Swift compiler features, the Swift repository’s README recommends building with --release-debuginfo --debug-swift, which includes debug information when compiling llvm, the Swift compiler, and the Swift standard library. A build like this would require at least 8 GB of RAM.

However, we can reduce the RAM required – and thereby adopt a less expensive server configuration – by building just the compiler part with debug symbols, with --release --debug-swift. Doing that would enable us build Swift on a 4 GB RAM machine. As long as we’re only going to work on compiler features, this build shall suffice.

In terms of disk space, if we build only Swift with debug symbols, we’ll need about 30 GB per build; if we build everything with debug symbols, we’ll need about 80 GB per build.

Apart from memory and diskspace, it’s a good idea to pick a server close to our location to reduce latency.

Once the Linux server is created and booted, we can start setting it up.

Set up the remote Linux machine

Set up a user

The freshly created server only has a root login now. We can login to that machine using the root password specified or obtained while creating the VPS:

$ ssh root@<ip-address> # Enter password

Once logged in, create a new user. Specify a password when prompted. You will need this password later when you use sudo commands.

# adduser <user> # Specify and remember password. Other info can be empty.
# usermod -aG sudo <user> # Give <user> sudo privileges
# su <user> # Switch to that user

Set up SSH keys

Create the .ssh directory:

$ mkdir ~/.ssh
$ chmod 700 ~/.ssh

Assuming you have an SSH keypair handy (they should be in ~/.ssh on your local machine), add your public key as an authorized key on the remote server.

$ cat >> ~/.ssh/authorized_keys # Paste your public key
$ chmod 600 ~/.ssh/authorized_keys

This should enable you to ssh into the remote server without having to type the server password.

Now, it’s a good idea to disable password-based login:

$ sudo vim /etc/ssh/sshd_config # set 'PasswordAuthentication' to 'no'
$ sudo systemctl reload sshd # Reload the SSH daemon

Set up tmux

We use tmux to keep terminal sessions alive on the server even when our SSH connection breaks. tmux is a terminal multiplexer, so one tmux session can encapsulate multiple terminal sessions.

Install tmux on the server, if it’s not installed already.

$ sudo apt-get install tmux

Create a new tmux session called “swift” in detatched mode.

$ tmux new-session -s swift -d

The newly created tmux session will have one tmux window, and therefore one terminal session. We can list the active tmux sessions by saying:

$ tmux list-sessions

We’ll see that we have one tmux session with one window. We can create more tmux windows and thereby, more terminal sessions. There are tmux commands to do that, but an easier way to work with tmux is through iTerm2 from your Mac.

Set up the local macOS machine

Set up iTerm2

If you don’t have iTerm2 on your Mac, download and install it. The following instructions assume iTerm2 version 3.3.7.

1. Open iTerm2

2. Go to iTerm2 > Preferences > Profiles and add a new profile (say “Swift dev”) using the “+” at the bottom left.

3. Select that profile on the left, then select the General tab on the right.

   Under the Command section, select Command and enter the command as:

   ssh -t <user>@<ip-address> "tmux -CC attach -t swift"
   

   substituting your remote server username and ip address.

Using iTerm2’s tmux integration

Once we’ve created the iTerm2 profile, if we right-click on iTerm2 on the dock, and click on New Window … > Swift dev (or whatever you named your profile), iTerm2 will:

• Open a new window and run the ssh command
• Once the ssh connection is established, run the tmux attach command on the server to attach to the tmux session we’d started
• Get the state of the tmux windows in the tmux session and show them as new tabs or windows in iTerm2

The tmux session we had created on the server has just one window. So if we right-click on iTerm2 and click on New Window … > Swift dev, two new windows shall open:

1. First, a window that runs ssh and then the tmux attach command. This window shall have no command prompt.
2. Next, a window that shows the tmux session’s tmux window, with a command prompt.

With the second window selected, iTerm2’s New Window and New Tab commands will show a prompt asking if we want to create a new tmux tab or window. If we pick that option, iTerm2 will create a new tmux window in our tmux session and will open that window in a new tab / window. When closing such a tab / window, iTerm2 will ask us if we want to kill that window in the tmux session, or do we want to just hide it. I prefer killing a tmux window when the corresponsing tab is closed.

If you’re done working on the server, you can close the first window, which shall close the ssh connection and will then close all the tmux windows. This can also happen when the ssh connection is broken due to inactivity or a broken network connection.

To get back to working on the server, you can right-click on iTerm2 and click on New Window … > Swift dev again, and all our tmux windows will be back.

Configuring iTerm2’s tmux integration

This is how I like my iTerm2 configured for working on Swift:

• Make each tab correspond to a tmux window without showing the prompts mentioned earlier

  Go to iTerm2 > Preferences > Advanced, search for “tmux”.

  • Under Tmux Integration, turn on Suppress alert asking what kind of tab to use in tmux integration and Suppress alert asking what kind of window to open in tmux integration.

  • Under Warnings, turn on Suppress kill/hide dialog when closing a tmux tab and Suppress kill/hide dialog when closing a tmux window.

  With that, while working on a tmux window, ⌘-T and ⌘-N open new tmux windows, and closing a tab with a tmux window kills the tmux window.

• Use different-sized windows

  Previously, tmux required that all tmux windows in a session be resized to the size of the smallest client viewport. So when we have tmux windows from one tmux session spread across multiple iTerm2 windows, resizing one of those iTerm2 windows will resize the others as well.

  If you have tmux version 2.9 or later (to know the tmux version, run tmux -V on the server), we can fix this:

  • Go to iTerm2 > Preferences > Advanced. Under Experimental Features, turn on Allow variable window sizes in tmux integration.

  • Go to iTerm2 > Preferences > General and select the tmux tab. For Open tmux windows as:, select “Native windows”.

  I learnt about this option from here.

• Account for many tmux windows in a tmux session

  When connecting to a tmux session, if you have too many tmux windows (10 tmux windows by default), iTerm2 will show the “tmux dashboard” instead of opening the windows directly. We can then open the unopened tmux windows in tabs or windows from the dashboard.

  I find this limit of 10 to be too low, so I set it to 20. This can be configured in the tmux dashboard itself, by going to Shell > tmux > Dashboard in the iTerm2 menu.

• Color scheme

  By default, the tmux windows are created with a profile called “tmux”. I changed the color scheme for this profile by doing this:

  1. Go to iTerm2 > Preferences > Profiles
  2. Select tmux on the left, select the Colors tab on the right
  3. Select a preset in Color Presets (I recommend Solarized Dark)

• Always show tab bar

  Even if there’s just one tab in a window, I like the tab bar to show up so that I can easily rearrange tmux windows into iTerm2 windows as I want. To do that:

  1. Go to iTerm2 > Preferences > Appearance
  2. Go to the Tabs tab
  3. Turn on Show tab bar even when there is only one tab

If you want to poke around for more tmux-specific configuration options, you can try these:

• See the iTerm2 wiki page on configuring iTerm2’s tmux integration
• Go to iTerm2 > Preferences > General and select the tmux tab
• Go to iTerm2 > Preferences > Advanced and search for “tmux”

Build Swift

Get the dependencies

On the server, install all the required dependancies to build Swift using the apt-get command from the Swift repository.

Get the code

Before building Swift, we need to clone Swift repository, and a bunch of other repositories as well. We create a directory under which all these repositories will be placed:

$ mkdir ~/swift-source

And then clone all the required repositories under that directory:

$ cd ~/swift-source
$ git clone https://github.com/apple/swift.git
$ ./swift/utils/update-checkout --clone

Build it

The Swift project’s build system uses CMake to create Ninja build files, and Ninja to use those build files to compile and link code. The Swift repository includes a build script called build-script that builds CMake and Ninja from source, and then uses that CMake and Ninja to build llvm, Swift and the Swift standard library.

By default, build-script will spawn as many parallel jobs as there are CPUs in the machine, which is good while compiling. However, linking uses a lot of memory, so we don’t want to be doing multiple link jobs at a time.

Update: To tell the build system not to use parallel jobs while linking, we can set the relevant CMake properties using the --llvm-cmake-options and --swift-cmake-options arguments.

To be able to work on compiler features, we want to build the compiler in debug mode. If you work on other parts of Swift (say the Swift standard library), or on other aspects of the compiler (like performance), the options would be different.

To build only Swift in debug mode and everything else in release mode, we say:

$ cd ~/swift-source
$ ./swift/utils/build-script --release --debug-swift --llvm-cmake-options==-DLLVM_PARALLEL_LINK_JOBS=1 --swift-cmake-options=-DSWIFT_PARALLEL_LINK_JOBS=1

This build can take about 3 hours, and will write the build results to build/Ninja-ReleaseAssert+swift-DebugAssert.

It helps to set up SWIFT_BUILD_DIR and PATH environment variables like this:

$ cat >> ~/.bash_profile
export SWIFT_BUILD_DIR=~/swift-source/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-linux-x86_64
export PATH=${PATH}:${SWIFT_BUILD_DIR}/bin # For swift, swiftc, etc.
export PATH=${PATH}:~/swift-source/llvm-project/utils/lit # For lit.py
$ source ~/.bash_profile

Having swift and swiftc in our PATH would help us run the built Swift from any directory. Having lit.py in our PATH would help us run the Swift testsuite from any directory.

In case you want to build everything with debug information (and your server has 8 GB RAM or more), you can pass --release-debuginfo instead of --release to build-script. This build will take longer to complete, and the build results will be placed in a different directory.

Set up the development environment

Git

Setup username and email for creating commit messages.

git config --global user.name “Roopesh Chander”
git config --global user.email roop@roopc.net

Setup the editor for editing commit messages and interactive rebase. I use vim.

git config --global core.editor vim

Vim

The Swift compiler’s C++ code uses two spaces for indentation, so we can add these to our ~/.vimrc:

set expandtab
set tabstop=2
set shiftwidth=2

For help in editing Swift, SIL and gyb files, we can use the vim support files from the Swift repository itself by:

• adding this line to ~/.vimrc:

  set runtimepath+=~/swift-source/swift/utils/vim
  

• making the filetype detection files accessible under ~/.vim/ftdetect:

  $ mkdir -p ~/.vim/ftdetect
  $ cd ~/.vim/ftdetect
  $ ln -s ~/swift-source/swift/utils/vim/ftdetect/*.vim .
  

lldb

We just need to install lldb:

$ sudo apt-get install lldb

Set up our fork of Swift

Contributions to the Swift compiler are done through GitHub pull requests. To create pull requests, we should have a fork of the Swift repository under our GitHub account. To fork Swift, go to Swift’s GitHub page on a web browser and click on the “Fork” button.

We already have a clone of the main Swift repository. We want to be able to push from there to our fork on GitHub. We don’t want to be specifying our GitHub password every time we push, so we’d like to authenticate to GitHub automatically using SSH keys. However, if we make our private keys available on the server, either by copying our private keys or by using SSH forwarding, it means that in case the server is compromised, that key can be used to control everything in our GitHub account. So, in the interest of security, we’re going to create a separate keypair that we shall use to push to just one repository – our fork of Swift. Since this key controls just this one repository, we can even afford to make the key not password-protected.

1. On the server, create a keypair:

   $ ssh-keygen
   

   and just press enter for the prompts (no password required). This shall create ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.

2. Start the SSH agent and add the private key to it:

   $ eval `ssh-agent -s`
   $ ssh-add ~/.ssh/id_rsa
   

3. Copy the public key:

   $ cat ~/.ssh/id_rsa.pub # Copy the output
   

4. Add this public key as a Deploy key in GitHub:

   • Go to your fork’s GitHub page on a web browser (the URL should be like https://github.com/<your-github-username>/swift) and go to Settings > Deploy keys > Add deploy key
   • Under Title, specify a title so that we can identify the deploy key’s origin in the future
   • Under Key, paste the public key we copied in the previous step
   • Tick the checkbox to Allow write access
   • Click on “Add key”

5. Back on the server, go to our clone of the Swift repository and add our fork as another remote called fork:

   $ cd ~/swift-source/swift
   $ git remote add fork git@github.com:<your-GitHub-username>/swift.git
   

At this point, we have a working setup. If you’re new to Swift compiler development, now would be a good time to start looking at starter bugs. In the next post, we’ll see how we can work on the Swift compiler with a setup like this.

If the server reboots, we’ll lose the tmux sessions. In that case, we have to start a fresh tmux session. We should also restart the SSH agent and ssh-add the private key for pushing to our fork.

Keep in mind that the server is a recurring cost. So if we know we’re going to not work on the compiler for a few weeks, it can be a good idea to delete the server and spawn a new one when we’re ready. Before deleting, we should take a look at all local branches and the stash, push anything worth saving to our fork, and file-transfer any uncommitted unit testcases, so that we can get back to work with minimum disruption.

If you have questions, corrections, or feedback, please get in touch.

Citron is based on the Lemon parser generator by Richard Hipp that’s used in SQLite. The parser generation engine of Citron is written in C, as is the case with Lemon, but the output parser code is all Swift.

What’s different

Citron uses the same grammar specification syntax used by Lemon, which is a little different from what Bison and Yacc use. In addition, Citron is stricter with code blocks and types than Bison, Yacc or Lemon.

In Citron:

• All terminals and non-terminals in the grammar must have explicitly defined types

• Every grammar rule should be accompanied by an associated code block that will be invoked when the rule gets used during parsing

• The code block associated with a rule should take in values of the correct type as input, and should return a value of the correct type

Writing a rule

Let’s say you have a grammar rule for Swift’s let statement that can be expressed in BNF like this:

<let-decl> ::= "let" <id> ":" <type-decl>

An equivalent rule in a Citron grammar file could look like this:

let_decl ::= LET ID COLON type_decl.

In Citron, terminals (a.k.a. tokens) start with an uppercase letter (like LET, ID and COLON), and non-terminals start with a lowecase letter (like let_decl and type_decl). “let”, “:” and the identifier are expected to be recognized at tokenization stage and converted to tokens LET, COLON and ID respectively, so the parser only sees these tokens.

If one were to create a parse tree from the input, there would be semantic types representing the terminals and non-terminals present in the grammar. A let node of the parse tree could be represented with something like:

class LetDeclaration {
    var id: String
    var type: TypeDeclaration
    init(id: String, type: TypeDeclaration) {
        self.id = id
        self.type = type
    }
}

If that’s the case, you should tell Citron about these backing types. It’s a good practice to declare the type of the terminals before the rules, and the declare the types of the non-terminals close to where the they occur in a rule, like this:

%token_type String // type representing LET, ID and COLON

%nonterminal_type let_decl LetDeclaration
%nonterminal_type type_decl TypeDeclaration

let_decl ::= LET ID COLON type_decl.

Citron expects a code block immediately after a rule. The code block should form the body of a function that returns a value representing the LHS symbol of the rule. The LHS symbol of this rule is let_decl, so our code block should return something of type LetDeclaration.

We can add alias names to the RHS symbols as shown below, and they will be made available as named function parameters within the code block with the appropriate type.

%token_type String
%nonterminal_type let_decl LetDeclaration
%nonterminal_type type_decl TypeDeclaration

let_decl ::= LET ID(n) COLON type_decl(t). {
    // 'n' of type String and 't' of type TypeDeclaration
    // are available for use in this code block
    return LetDeclaration(id: n, type: t)
}

Now we have a valid Citron rule – the rule is immediately followed by a code block, and the code block takes in and returns values of correct types.

You can find an example grammar file here, which has a set of rules that describe simple arithmetic expressions.

In the Citron-generated code, the code block is used as the body of a function. The type signature of the function is based on the symbol on the LHS and the aliased symbols on the RHS. For the above code block, the Citron-generated function would look something like this:

func codeBlockForRule1(n: String, t: TypeDeclaration) -> LetDeclaration {
    // 'n' of type String and 't' of type TypeDeclaration
    // are available for use in this code block
    return LetDeclaration(name: n, type: t)
}

Unless the aliased symbols’ types and the return type used in the code block are consistent with the rule, the generated parser code will not compile. This type safety helps catch code block errors at build time rather than at run time.

The parser interface

The Citron-generated code contains a parser class, named Parser by default, with two parsing methods:

1. consume(token:code:)

   This methods makes the parser consume one token. This should be called multiple times to pass a sequence of tokens to the parser. Typically, a separate tokenization stage would generate this sequence of tokens. As the tokens get generated, they can be passed to the parser through this method.

   The first argument, token:, is the semantic value of the token, as seen by the code blocks in the grammar. The type of this argument is the %token_type type specified in the grammar.

   The second argument, code:, is the token code that Citron knows this token by. The Citron-generated class contains an enum of all terminals in the grammar, and this argument should be a value of that enum.

   For example, the “n” in “let n: Int” would be passed to the parser like this: parser.consume(token: "n", code: .ID).

2. endParsing()

   This method tells the parser that there are no more tokens to consume, and signifies the end of input.

   This method takes no arguments, and returns the parse result. The parse result is the value returned by the code block of the start rule (a.k.a. root rule) of the grammar. If we’re building a parse tree using the code blocks as illustrated above, the parse result would typically be the parse tree.

   The return type of this method is the %nonterminal_type of the start symbol (a.k.a. root symbol) of the grammar.

The string “let n: Int” could get tokenized into four tokens and passed to Citron as:

let parser = Parser()
do {
    ...
    try parser.consume(token: "let", code: .LET)
    try parser.consume(token: "n", code: .ID)
    try parser.consume(token: ":", code: .COLON)
    try parser.consume(token: "Int", code: .BUILTIN_TYPE)
    ...
    let parseTree = try parser.endParsing()
} catch {
    // Handle errors
}

Error handling

Both consume(token:code:) and endParsing() are throwing methods, so they should be invoked with a preceding try. They throw when an input token at a certain position is inconsistent with the grammar, or when the input ends prematurely.

Moreover, we can throw errors from within a rule’s code block and those throws would propagate up to one of the two parsing methods.

Update: In Citron 2.0, you can use error capturing for more advanced error handling.

The lexer interface

Citron offers a simple lexer that can tokenize an input string.

We give the lexer a series of rules with either string or regular expression patterns. The token data that the lexer should output can be of an arbitrary type, so the lexer is defined as a generic type, with the token data as a type parameter. For each string pattern, we give it the token data that should be output, and for each regular expression pattern, we give it a closure that returns the token data.

For example, you could say:

// A tuple with all the data we need for a token
typealias TokenData = (String, Parser.CitronTokenCode)

// A lexer that can output tokens as the tuple we just defined
typealias Lexer = CitronLexer<TokenData>

// Create the lexer with tokenization rules
let lexer = Lexer(rules: [

        // Keywords

        .string("let", ("", .LET)),
        .string("var", ("", .VAR)),
        .string("class", ("", .CLASS)),
        ...

        // Punctuation

        .string("{", ("", .OPEN_FLOWER_BRACKET)),
        .string("}", ("", .CLOSE_FLOWER_BRACKET)),
        .string(":", ("", .COLON)),
        ...

        // Built-in types

        .string("Int", ("", code: .INT)),
        .string("String", ("", code: .STRING)),
        ...

        // Identifiers

        .regexPattern("[A-Za-z][0-9A-Za-z]*", { str in (str, .ID) }),

        // Whitespace

        .regexPattern("\\s", { _ in nil }) // Ignored
    ])

We can ask the lexer to tokenize a string and give it a closure to run on each token it identifies. The token data gets passed into the closure as a parameter. Inside the closure, we can pass on the token data to the parser.

do {
    try lexer.tokenize(inputString) { (t, c) in
        try parser.consume(token: t, code: c)
    }
    let parseTree = try parser.endParsing()
    print("\(parseTree)")
} catch {
    print("Error during parsing")
}

Try it

The Citron code is on Github, along with a few examples. There’s also a more detailed documentation.

I’d be happy to answer questions or take feedback on Twitter or by email.

Telecom regulator Trai has accused iPhone maker Apple of engaging in “data colonisation” in India and being “anti-consumer” by not allowing customers to pass on details about pesky calls and unwanted messages to authorities as well as their mobile operators.

…

“While Google’s Android supports our Do-Not-Disturb (DND) app, Apple has just been discussing, discussing, and discussing. They have not done anything,” Sharma told TOI.

Sharma is the Chairman of the Telecom Regulatory Authority of India (TRAI), a government organization that, among other things, oversees the functioning of mobile network carriers in India.

TRAI has published an Android app called Do-Not-Disturb to help curb marketing cold calls and SMS messages. The app is tied to TRAI’s NCPR database of mobile users and their opt-in / opt-out preferences regarding receiving promotionals calls and SMS messages. A key feature of the app is helping the user report a promotional call or SMS they have received that doesn’t comply with their registered preference. To implement this feature, the app requires access to the phone’s call logs and messages. It displays recent calls and messages that the user can pick from and report a complaint on.

When the Android app was launched last year, TRAI did have plans of bringing a similar app to iOS as well, but that hasn’t happened so far. A Public Interest Litigation had been filed in the Delhi High Court asking for such an app to be made available on the App Store, and was dismissed by the court last month.

There’s no public API in iOS to read call logs or SMS messages. There are ways to block certain callers and filter certain SMS messages ¹, but the provided API is restricted because of privacy considerations. To block callers, apps can provide a block list but have no way to intercept calls or access the call log. To filter SMS messages, apps can classify SMS messages from unknown senders as they come, but are not allowed to keep track of them and use them later on (like showing a list of messages to choose from).

What TRAI could do

TRAI could, with the current available API in iOS, write an app that could be useful in reporting spam calls by including a share extension for contacts ².

The user, while viewing the call log on an iPhone, can tap the info button and then the ‘Share Contact’ button to bring up a share sheet that could show a TRAI share extension to report complaints. The share extension would get the phone number from the shared contact information, but there’s no way to share the date the spam call was made. However, there are only three possible options: ‘Today’, ‘Yesterday’ and ‘Two days ago’ (because TRAI requires that the complaint be filed within 3 days), so it should be okay to ask the user for that information.

So that would give us a pretty good way to report spam calls.

This share extension method would also work for reporting promotional SMS messages received from actual phone numbers, but would unfortunately not work for SMS messages with alphanumeric sender IDs. That means in most SMS reporting scenarios, this wouldn’t be useful.

What Apple could do

I think Apple is highly unlikely to create public APIs to access call logs or messages, but that’s not really needed for an app like this.

Rather, Apple could provide a way to share an SMS through a share sheet, so a third-party app can get access to all relevant fields of an SMS message (i.e. body text, sender id and timestamp) that the user explicitly chose to share with the app. That way, TRAI can create an app to report errant SMS messages, while Apple can retain their right to not give apps blanket access to the phone’s SMS data.

Update: Apple now offers a much more complete solution for reporting spam calls and SMSes with the SMS and Call Spam Reporting API, which would be perfect for a TRAI iOS app to make use of.

So this is how I think TRAI and Apple can settle this without compromising either party’s positions, and simultaneously keep Apple’s privacy-conscious customers happy as well.

Foremost, the App Accelerator seems to be a sort of permanent Apple Tech Talks venue. The first talks started yesterday, and the schedule for the next few weeks covers mostly introductory topics, leaning towards iOS (UI Design, Swift, Adaptive UI, Accessibility, i18n, etc.). All slots for all scheduled presentations were already filled up when I took a look last Friday (when the news broke). At present, the talks seem to be only on Tuesdays and Thursdays, so there’s plenty of room for more. I’d love to see in-depth talks on specific topics (say Core Data, CloudKit or UIKit Animations). Phil Schiller mentioned that the App Accelerator is also going to help developers in “learning how better to market their app on the App Store”, so there could be talks on that topic as well, which would be very nice.

Apart from the talks, the website says there are going to be hands-on labs, but no schedule is out yet.

Moreover, the App Accelerator could also serve as a place for Apple to sit down and talk to some developers of apps it considers important. The Apple announcement mentions Practo and Reliance Games, and it looks like the Legend of Abhimanyu team visited the App Accelerator last week.

All in all, this is great news – I’d find the talks and the labs very useful.

I also think it shows how serious Apple is about a long-term play in the Indian market. Phil Schiller, when talking to NDTV last week, said “I think what we hope from this accelerator is that we can help the local market create apps for customers in India that better meet the needs of our growing customer base here”. There are tons of India centric apps on Android, but not many on iOS, which is a reflection of the Indian smartphone market being heavily skewed towards Android. Getting more India-centric apps would be an important cog in Apple’s wheel for growing bigger in India.