Linux Support #1188

art-divin · 2023-07-30T19:10:08Z

Resolves #306

Description

This MR enables support on Linux (particularly was compiled under Ubuntu 22.04, but should work on other version as well if all dependencies are met).

Technical Details

A couple of things were disabled (#1198):

Due to the fact that FileWatcher relies on macOS SDK and uses FSEvents framework, it was disabled, i.e. folder watching flag (--watch) is not working under Linux
All JavaScriptCore functionality (ejs templates etc.) were disabled under Linux only
NSObject available under Linux does not conform to KVC, thus tests in TypedSpec.generated.swift were moved to class definitions.

Notes

getVaList was replaced with recommended withVaList() {} (used mostly for Codable-related code generation)
NSException was replaced with fatalError only under Linux
CFAbsoluteTimeGetCurrent was replaced with Date().timeIntervalSince1970 for all platforms
FolderWatcher needs to be implemented probably using approach similar to this one
sha256() of Data extension is implemented via swift-crypto package

Environment Setup

I have installed ubuntu VM through tart and updated to 22.04 according to this guide.

I had to run the following commands prior to being able to run bundle install in Sourcery:

sudo apt install libffi-dev
sudo apt install build-essential
sudo apt install libsqlite3-dev
sudo apt-get install libncurses5-dev

SourceryBot · 2023-07-30T19:13:26Z

	1 Warning
⚠️	Big PR

Generated by 🚫 Danger

fixed wrong find-replace

art-divin · 2023-08-06T15:05:47Z

I was able to make tests green on macOS while preserving successful compilation under Linux 🔥

However, swift test does not run tests under Linux, i.e. it compiles for testing, and swift test list also generates plenty of discovered tests under .build/linux..../debug/*DiscoveredTests, but it does not execute any tests.

I also need support how to setup CircleCI to run linux and also compile Sourcery for releases under Linux @krzysztofzablocki do you have an insight? Maybe a simple hint, anything would be appreciated 👍🏻

🙏🏻

krzysztofzablocki · 2023-08-07T08:08:48Z

I don't really have experience with Linux but you could also move us to Github actions if you prefer, this project predates them but we could easily use GH actions now

art-divin · 2023-08-07T20:46:23Z

Thanks @krzysztofzablocki .

Apart from all Linux-related stuff, I have faced an issue when running tests with swift test. Seems like bundle is different when swift test is running tests (creating xctest file) and when Xcode does the same (generates xctest file) ->

When swift test is executed, the same project generates SourceryPackageTests.xctest bundle, which does not contain resources,
while when Xcode runs the same tests, bundle SourceryLibTests.xctest is generated, and that bundle contains described resources as per Package.swift spec.

Looking into it, but this is closely related to why .ejs are disabled for SPM-compiled sourcery (related to #244)

art-divin · 2023-08-07T21:16:09Z

Tried to rename SourceryLibTests to SourceryPackageTests just to make swift test "think" that it's OK to put resources there, and yet it did not help. Here's the difference:

My suspicion is that swift test does not support copying resources just yet, found something related to this here.

Update: found the relevant Swift forums thread here. I'll try the workaround, once successful, swift test would be used for running Sourcery tests.

art-divin · 2023-08-07T21:38:31Z

My current plan is:

Make swift test work
Enable GitHub Actions to compile for macOS and Linux
Make tests executable under Linux
Merge this MR

Implemented GH actions for macOS and ubuntu; disabled both actions until #1188 is merged.

…nux-support

art-divin · 2023-09-04T16:28:14Z

Sourcery/Sourcery.swift

        return try templatePaths(from: from).compactMap {
            if $0.extension == "sourcerytemplate" {
                let template = try JSONDecoder().decode(SourceryTemplate.self, from: $0.read())
                switch template.instance.kind {
                case .ejs:
-                    guard EJSTemplate.ejsPath != nil else {
-                        Log.warning("Skipping template \($0). JavaScript templates require EJS path to be set manually when using Sourcery built with Swift Package Manager. Use `--ejsPath` command line argument to set it.")


@krzysztofzablocki here I was really unsure about this guard statements. Due to changes I have introduced in EJSTemplate.swift file in this MR, swift test started to work, but for tests, not for a release build which I am unsure how to verify.

Could you please share some insight on this, shall I leave these guard statements as they were, or would it actually work due to this change I have mentioned?

Thank you 🙏🏻

if we don't have the EJS contents then the templates wouldn't work but you could also reject it based on env since you mentioned JavascriptCore doesn't work on Linux

Nice one, thanks. I'll revert these changes to throw a more precise error as before.

Newer SPM (I think 5.9) supports "resources embedded into binary executables" (see .embedInCode) - maybe we could include EJS template contents into code (encoded, say, with base16/64 encoding or as array of bytes) - I'd need to dig deeper into this, but sounds like some form of convenience for Sourcery. I'll revisit this later.

art-divin · 2023-09-04T16:29:36Z

SourceryRuntime/Sources/AST/AssociatedValue_Linux.swift

+/// Defines enum case associated value
+public final class AssociatedValue: NSObject, SourceryModel, AutoDescription, Typed, Annotated, Diffable, DynamicMemberLookup {
+    public subscript(dynamicMember member: String) -> Any? {
+        switch member {


This list is not exhaustive, but just to make all unit tests work. It must be autogenerated like init(coder:) methods, and I'll address it in #1198

art-divin · 2023-09-04T16:30:42Z

SourceryRuntime/Sources/AST/Enum_Linux.swift

+import Foundation
+
+/// Defines Swift enum
+public final class Enum: Type {


Due to limitations how #if statement can be used in Swift, I figured it is better to have separate files rather than having multiple definitions in already overcrowded files.

art-divin · 2023-09-04T16:33:32Z

SourceryTests/Generating/StencilTemplateSpec.swift

+                context("given array") {
+                    it("doesnt modify the value") {
+                        let result = generate("{% for key,value in type.MyClass.variables.2.annotations %}{{ value | toArray }}{% endfor %}")
+                        expect(result).to(equal("[\"Hello\", \"beautiful\", \"World\"]"))


On linux, stringify from Stencil provides quotes around string values. I could not workaround it, and most likely this is due to differences in how NSObject works on two platforms.

art-divin · 2023-09-04T16:34:15Z

SourceryTests/GeneratorSpec.swift

                it("can render variable isOptional") {
                    expect(generate("{{ type.Complex.variables.first.isOptional }}")).to(equal("0"))
                }
+                #else
+                it("can render variable isOptional") {
+                    expect(generate("{{ type.Complex.variables.first.isOptional }}")).to(equal("false"))


🤷🏻 Don't ask me why 🤷🏻 Really weird difference

art-divin · 2023-09-04T16:35:35Z

SourceryTests/SourcerySpec.swift

@@ -1271,14 +1294,15 @@ class SourcerySpecTests: QuickSpec {
                        }.toNot(throwError())
                }

+#if canImport(ObjectiveC)


This is related to crash in Swift compiler reported here.

art-divin · 2023-09-04T16:36:19Z

Templates/Tests/TemplatesTests.swift

@@ -82,6 +90,12 @@ class TemplatesTests: QuickSpec {
            expect(generatedFileFilteredLines).to(equal(expectedFileFilteredLines))
        }

+#if !canImport(ObjectiveC)


On Linux, static func setUp() is not called, so I needed to manually add this beforeSuite to generate files before tests are run.

art-divin · 2023-09-04T16:38:03Z

Templates/Tests/TemplatesTests.swift

+        #if canImport(ObjectiveC)
+        let contextSources = "\(resources)/Context"
+        #else
+        let contextSources = "\(resources)/Context_Linux"


Different file folders are used for two reasons:

SPM does not have a method to specify where to put the copied resources (Resource.copy(path:locatization:)).

Because of the crash in Swift compiler, AutoCodable.swift file needs to be omitted when tests are run on Linux

Joannis · 2023-09-04T18:33:21Z

AMAZING. Looking forward to seeing this hopefully merged

krzysztofzablocki

Some small comments but amazing work 👏

krzysztofzablocki · 2023-09-07T08:57:22Z

Sourcery/Sourcery.swift

        return try templatePaths(from: from).compactMap {
            if $0.extension == "sourcerytemplate" {
                let template = try JSONDecoder().decode(SourceryTemplate.self, from: $0.read())
                switch template.instance.kind {
                case .ejs:
-                    guard EJSTemplate.ejsPath != nil else {
-                        Log.warning("Skipping template \($0). JavaScript templates require EJS path to be set manually when using Sourcery built with Swift Package Manager. Use `--ejsPath` command line argument to set it.")


if we don't have the EJS contents then the templates wouldn't work but you could also reject it based on env since you mentioned JavascriptCore doesn't work on Linux

SourceryFramework/Sources/Parsing/Utils/Time.swift

SourceryRuntime/Sources/AST/TypeName/Tuple_Linux.swift

work in progress

b626444

art-divin added 16 commits July 31, 2023 09:41

More conditional checks, less errors

4ab1b68

Further steps onto successful compilation

b1bad00

Enabled try-catch back for macOS

01ace5b

reverted redundant change

b1aed0b

Replaced getVaList with recommended withVaList

cc22cd2

Update FileParser + AttributesModifierSpec.swift

b8ceb95

fixed wrong find-replace

added missing import of CDispatch

6424fc7

enabled fatalError instead of NSException

9ccf240

enabled value(forKey:) for TypesCollection type

69d8117

removed commented import

299506a

enabled sha256 for macOS

5e280ad

Enabled sha256 hashing for Linux

d2d6835

disabled JavaScriptTemplate tests under Linux

525488e

added check for vendor for tests

4c22672

disabled value(forKeyPath:) calls under Linux

c3e4ef7

reverted redundant gem

b2390c4

art-divin requested a review from krzysztofzablocki August 6, 2023 15:04

art-divin changed the title ~~Draft: Linux Support~~ Linux Support Aug 6, 2023

art-divin added 3 commits August 10, 2023 21:45

Fixed issue with swift test not finding .ejs bundle

8a12dc5

removed commented code

6e507ff

switched to swift test

ba56bc8

art-divin mentioned this pull request Aug 12, 2023

Create swift.yml #1189

Merged

art-divin added a commit that referenced this pull request Aug 12, 2023

Create swift.yml (#1189)

7c04194

Implemented GH actions for macOS and ubuntu; disabled both actions until #1188 is merged.

art-divin added 5 commits September 4, 2023 13:05

Added AssociatedValue for Linux

2aebaf5

Added missing member lookups

ccd4adc

Changed resource copy path in TemplatesTests for Linux

4a709e7

Updated CHANGELOG

b589dd9

Merge branch 'linux-support' of github.com:art-divin/Sourcery into li…

14ccb2f

…nux-support

art-divin mentioned this pull request Sep 4, 2023

Linux Support: from experimental to 100% 🚀 #1198

Open

8 tasks

art-divin added 3 commits September 4, 2023 19:49

Added LINUX.md

21e181c

Added references to README and LINUX

89d065d

Updated content.generated files

4737bfe

art-divin commented Sep 4, 2023

View reviewed changes

art-divin mentioned this pull request Sep 4, 2023

Linux Support #306

Closed

art-divin added 2 commits September 4, 2023 23:12

Fixed a typo [no ci]

1afa3df

Changed wording related to @objc attribute

6c07f15

krzysztofzablocki approved these changes Sep 7, 2023

View reviewed changes

art-divin added 2 commits September 7, 2023 15:09

Use currentTimestamp() instead of Date().timeIntervalSince1970

adf54fa

Returned guard statements; removed default values from EJSTemplate

b286948

art-divin merged commit ba63cd6 into krzysztofzablocki:master Sep 7, 2023
3 checks passed

chenrui333 mentioned this pull request Sep 17, 2023

sourcery 2.1.0 Homebrew/homebrew-core#142581

Merged

ugoArangino mentioned this pull request Sep 18, 2023

Issue when running sourcery from binary #1202

Closed

art-divin mentioned this pull request Sep 18, 2023

Separate EJSTemplate.swift for swift test and for swift build -c release #1203

Merged

art-divin mentioned this pull request Jan 20, 2024

Road to basic Linux compatibility #630

Closed

art-divin mentioned this pull request Mar 11, 2024

Adjusted file structure to accommodate two generated files #1299

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux Support #1188

Linux Support #1188

art-divin commented Jul 30, 2023 •

edited

SourceryBot commented Jul 30, 2023 •

edited

art-divin commented Aug 6, 2023 •

edited

krzysztofzablocki commented Aug 7, 2023 •

edited

art-divin commented Aug 7, 2023

art-divin commented Aug 7, 2023 •

edited

art-divin commented Aug 7, 2023 •

edited

art-divin Sep 4, 2023

krzysztofzablocki Sep 7, 2023

art-divin Sep 7, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

art-divin Sep 4, 2023

Joannis commented Sep 4, 2023

krzysztofzablocki left a comment

krzysztofzablocki Sep 7, 2023

Linux Support #1188

Linux Support #1188

Conversation

art-divin commented Jul 30, 2023 • edited

Description

Technical Details

Notes

Environment Setup

SourceryBot commented Jul 30, 2023 • edited

art-divin commented Aug 6, 2023 • edited

krzysztofzablocki commented Aug 7, 2023 • edited

art-divin commented Aug 7, 2023

art-divin commented Aug 7, 2023 • edited

art-divin commented Aug 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Joannis commented Sep 4, 2023

krzysztofzablocki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

art-divin commented Jul 30, 2023 •

edited

SourceryBot commented Jul 30, 2023 •

edited

art-divin commented Aug 6, 2023 •

edited

krzysztofzablocki commented Aug 7, 2023 •

edited

art-divin commented Aug 7, 2023 •

edited

art-divin commented Aug 7, 2023 •

edited