IDF master

[me-no-dev]

esp-dsp: master 801c6dd
esp-face: master 420fc7e
esp-rainmaker: f1b82c7
esp32-camera: master 7c5d8b2
esp_littlefs: master b58f00c

[me-no-dev]

@valeros@ivankravets guys, toolchain has to be updated yet again :) Please think of a way that we can specify the necessary version in the pltformio build files. Toolchains are very much version dependent now.

[Jason2866]

@me-no-dev platformio could support directly the toolchains from espressif github (for testing stage version) if the would include a package.json.
The toolchain from earlephilhower for the esp8266 do include a package.json https://github.com/earlephilhower/esp-quick-toolchain. So it is possible with platformio to use latest Arduino Esp8266 stage and the needed latest new toolchain.

[me-no-dev]

@Jason2866 thanks for reminding me about this :) but it's not that straightforward to implement it on our end, because it's not us packing and compiling the toolchain. Files and binaries are signed and should not be unpacked and repacked by us (also makes little sense), but I will ask my colleagues to implement this for future versions. Is there a clear tutorial on what needs to be added? What should that json contain?

[Jason2866]

@me-no-dev thx for talking to your colleagues about this. I opened a features request a while ago for this and it was closed. I think i couldnt explain the needs why clearly enough.
Nothing fancy which is needed in the package.json. Only what it is, the target OS and a valid version number. Example from ArduinoEsp8266 toolchain (without the .txt extension)
package.json.txt
Platformio doc https://docs.platformio.org/en/latest/core/userguide/package/cmd_pack.html
The python code which parses the manifest file https://github.com/platformio/platformio-core/blob/develop/platformio/package/manifest/schema.py

[me-no-dev]

thanks @Jason2866 :)

@valeros, @ivankravets could you please clarify what would be the proper values for ESP32x toolchains?

{"description": "xtensa-gcc",//what would be the value for each ESP32 chip? C3 is RISCV"name": "toolchain-xtensa",//goes with the one above"system": [ "windows_amd64" ],//what are the options here?"url": "https://github.com/espressif/crosstool-NG", "version": "5.100300.210606"//how should we format the version? }

[ivankravets]

Hi there,

platformio could support directly the toolchains from espressif github (for testing stage version) if the would include a package.json.

It could potentially work for a "framework" type of package which is platform agnostic but will not work for a toolchain that should be precompiled for different architectures.

Files and binaries are signed and should not be unpacked and repacked by us (also makes little sense), but I will ask my colleagues to implement this for future versions

You can keep official packages without package.json. However, the only you package and sign them, you can generate package.json dynamically and publish them to a PlatformIO Trusted Package Registry.

---

PlatformIO Trusted Package Registry is a paid product for enterprises and is a part of our business model. We will contact the Espressif company and you will be able to publish packages directly from espressif organization.

[Jason2866]

@ivankravets imho best way to publish directly from espressif to platformio registry
By doing this Platformio users can always use latest development versions for IDF and Arduino ESP32. 👍
@me-no-dev i hope you can convience espressif crew to do this.
Without Platformio some nice projects would not be possible, and many active open source teams do use.

[Jason2866]

thanks @Jason2866 :)

@valeros, @ivankravets could you please clarify what would be the proper values for ESP32x toolchains?

{"description": "xtensa-gcc",//what would be the value for each ESP32 chip? C3 is RISCV"name": "toolchain-xtensa",//goes with the one above"system": [ "windows_amd64" ],//what are the options here?"url": "https://github.com/espressif/crosstool-NG", "version": "5.100300.210606"//how should we format the version? }

For completness: Example package.json for the "old" riscv toolchain for platform linux_x86_64

{"name": "toolchain-riscv-esp", "version": "1.80400.0", "description": "GCC Toolchain for Espressif 32-bit RISC-V based on GCC", "keywords": [ "build tools", "compiler", "assembler", "linker", "preprocessor", "espressif", "esp32", "risc-v" ], "license": "GPL-2.0-or-later", "repository":{"type": "git", "url": "https://github.com/espressif/crosstool-NG" }, "system": [ "linux_x86_64" ] } 

[igrr]

@me-no-dev I'll take a look at the linker error, it is unexpected.

I think the underlying issue is that CI in this repository relies on feature/idf-master branch of https://github.com/platformio/platform-espressif32 (

arduino-esp32/.github/scripts/install-platformio-esp32.sh

Line 4 in c37557c

PLATFORMIO_ESP32_URL="https://github.com/platformio/platform-espressif32.git#feature/idf-master"

) which currently can't build IDF master branch due to the new toolchain being required.

What do you think about updating to the tip of release/v4.3 IDF branch rather than master at this point? That should work with platformio's feature/idf-v4.3 branch here: https://github.com/platformio/platform-espressif32/tree/feature/idf-v4.3. Or is there something from IDF master branch that is necessary that is not present on release/v4.3?

[me-no-dev]

@igrr in all cases it will not fix the issue next time the toolchain needs to be updated (previous IDF versions get updates also). To switch to 4.3 will require unknown amount of changes here and in the lib builder, which is far more than updating the toolchain. I think the greater goal would be to somehow automate toolchain update and note the required version in the PIO build scripts. Sticking to 4.4 will also allow seamless addition of S3 support.

[me-no-dev]

@ivankravets I had a look at the PIO TPR and it seems that it's a storage solution. We do not need storage, our toolchains are available online already. Maybe I am not getting something?

[ivankravets]

@me-no-dev PlatformIO does not support untrusted package sources or 3rd party archives. It strictly works only with PlatformIO Trusted Package Registry where we guarantee package authenticity. Enterprises manage their packages in our registry and keep control of package distribution. We already contacted the Espressif company and you will receive full control of already published Espressif packages soon.

[me-no-dev]

@ivankravets this is PR CI build. It is not merged changes. If I am not able to specify the toolchain in the build files, but add the new version to your TPR, wouldn't that mean that all dev channel users will get that new version? Even though changes are not merged? Maybe we could specify toolchain path on the command line? But really the best case would be to be able to just specify the version in the build script :)

[Jason2866]

If the build chain is uploaded in platformio registry everyone can use it from platformio by specifying with the owner/version number. Uploading a package to registry is not depending on "something".
Before the C3 was supported from platformio i uploaded the needed riscv toolchains (linux, apple windows) for the C3 to the platformio registry. It can simply used in platformio with a entry in platform_packages -> tasmota/toolchain-riscv32 @ ~1.80400.210415
The complete setup can be user made. In the used example below the platform and all platform_packages where selfmade :-)

platform = https://github.com/Jason2866/platform-espressif32.git#feature/arduino-c3 platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/291/framework-arduinoespressif32-master-1d7068e4b.tar.gz tasmota/toolchain-riscv32 @ ~1.80400.210415 

So i think you can specify with small changes in your build script the toolchain you want / need.
If owner espressif is uploading the toolchains everyone can use with espressif/toolchain-riscv32 @ ~version.date
Best solution you provide a platform.json in your github (for the different branches) via a fork from https://github.com/platformio/platform-espressif32
everyone could always use latest version with platformio and your changes in build script would be minimal!
Just this entry would be needed in platformio setup:
platform = https://github.com/espressif/platform-espressif32.git#idf-master
as example for the branch idf-master
Maintain of platform.json is easy and i am sure you can automate via scripting.
@me-no-dev

[me-no-dev]

@Jason2866 what would b the default toolchain if not specified?

[me-no-dev]

thanks for the extensive explanation @Jason2866 I'm starting to get things now.
When you uploaded the toolchains, did you have to add platform.json into the archives?

Let's say that we fork platformio/platform-espressif32 to espressif/platform-arduinoespressif32 and include the following platform.json

{"name": "arduinoespressif32", "title": "Espressif 32", "description": "Espressif Systems is a privately held fabless semiconductor company. They provide wireless communications and Wi-Fi chips which are widely used in mobile devices and the Internet of Things applications.", "homepage": "https://espressif.com/", "license": "Apache-2.0", "keywords": [ "dev-platform", "Wi-Fi", "Bluetooth", "Xtensa", "LX6", "riscv" ], "engines":{"platformio": "^5" }, "repository":{"type": "git", "url": "https://github.com/espressif/platform-arduinoespressif32.git" }, "version": "3.2.0", "frameworks":{"arduino":{"package": "framework-arduinoespressif32", "script": "builder/frameworks/arduino.py" } }, "packages":{"toolchain-riscv32":{"type": "toolchain", "optional": true, "owner": "espressif", "version": "~1.80400.1234" }, "toolchain-xtensa32":{"type": "toolchain", "optional": true, "owner": "espressif", "version": "~2.80400.1234" }, "toolchain-xtensa32s2":{"type": "toolchain", "optional": true, "owner": "espressif", "version": "~1.80400.1234" }, "framework-arduinoespressif32":{"type": "framework", "optional": true, "owner": "espressif", "version": "https://github.com/espressif/arduino-esp32.git" }, "tool-esptoolpy":{"type": "uploader", "owner": "platformio", "version": "~1.30100.0" }, "tool-mkspiffs":{"type": "uploader", "optional": true, "owner": "platformio", "version": "~2.230.0" } } }

With that anyone can build projects with the following example config:

[env:esp32dev]platform = https://github.com/espressif/platform-arduinoespressif32.git framework = arduino board = esp32dev monitor_speed = 115200

Then let's say that we have a branch called idf-master, where we need to add a new toolchain version.

1. We upload the new toolchain to platformio
2. Edit platform.json in that branch to update the version numbers (ex. ~1.80400.1234 to ~1.80400.5678)
3. Run CI specifying the new platform (platform = https://github.com/espressif/platform-arduinoespressif32.git#idf-master)
4. Or specify just the new toolchain (platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.5678)
5. Merge the changes in both platform-arduinoespressif32 and arduino-esp32 at the same time

@Jason2866 does that sound any good?
If we can specify URL for version, then could we say platform_packages = espressif/toolchain-xtensa32 @ https://github.com/espressif/crosstool-NG/releases/download/esp-2021r1/riscv32-esp-elf-gcc8_4_0-esp-2021r1-linux-amd64.tar.gz?

[Jason2866]

@me-no-dev great you are considering to do a complete working platfomio setup!

When you uploaded the toolchains, did you have to add platform.json into the archives

No, only the valid package.json for this toolchain is needed. So you have to upload the toolchain one by one for every OS
A adopted platform.json is needed in your fork of https://github.com/platformio/platform-arduinoespressif32.git
You create branches (easiest same name as your branch names from https://github.com/espressif/arduino-esp32)
In this branches you change the platform.json accordingly to your needs of the referenced namend branch.

1. We upload the new toolchain to platformio

Always needed / correct

2. Edit platform.json in that branch to update the version numbers (ex. ~1.80400.1234 to ~1.80400.5678)

Correct too, to do in your forked and namend branch of https://github.com/platformio/platform-arduinoespressif32.git

3. Run CI specifying the new platform (platform = https://github.com/espressif/platform-arduinoespressif32.git#idf-master)

100% correct

4. Or specify just the new toolchain (platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.5678)

100% correct too

5. Merge the changes in both platform-arduinoespressif32 and arduino-esp32 at the same time

Only needed when changes in toolchain or needed tools referenced in platform.json

@Jason2866 does that sound any good?

Yes, sounds good. If you try one time you will see it is easier than then whole explanations :-)

If we can specify URL for version, then could we say platform_packages = espressif/toolchain-xtensa32 @ https://github.com/espressif/crosstool-NG/releases/download/esp-2021r1/riscv32-esp-elf-gcc8_4_0-esp-2021r1-linux-amd64.tar.gz?

Yes, this is possible (and that what i suggested before) WHEN in this tarball is a valid package.json for linux-amd64.
As Ivan wrote this toolchain is only working for linux-amd64 For CI it is totally okay since it is always Linux.
A windows user needs to reference the correct windows variant. So thats why it is the best solution to upload all variants to platformio registry. By doing this it is OS independend. The entry would look like this platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.210630
This is the way to provide the toolchains for everyone to use with platformio

[Jason2866]

@Jason2866 what would b the default toolchain if not specified?

The default toolchain is the referenced one in platfom.json

[ivankravets]

@me-no-dev you don't need to fork the original ESP32 dev-platform for PlatformIO. The steps are very simple:

1. Publish a new toolchain using espressif organization to the PlatformIO Registry.
2. Patch platform.json in https://github.com/espressif/arduino-esp32/blob/master/.github/scripts/install-platformio-esp32.sh with your custom toolchain version (we will help here)

P.S: We are in progress with creating espressif organization in PlatformIO Registry. The only we finish it, we will back here.

[Jason2866]

@ivankravets@me-no-dev Any news? We would like to use the new toolchains with Platformio.
Thx!

[Jason2866]

Thx @me-no-dev@ivankravets@valeros
Already using :-)