diff --git a/.gitignore b/.gitignore index beba5119..a6c7bcd2 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ /BUILD-ARTIFACTS/ /Makefile TAGS +/bitbake-cookerdaemon.log /bitbake.lock /mcf.status /oe-init-build-env @@ -30,6 +31,8 @@ TAGS __pycache__/ tmp/ patches/ +/oe-logging.json +/ccache_stats.json # Artifacts from generating build_changes.log build_changes.log diff --git a/README.md b/README.md index a698b9be..af52881b 100644 --- a/README.md +++ b/README.md @@ -1,101 +1,65 @@ -build-webos -=========== +# build-webos -Summary -------- -Build webOS OSE (Open Source Edition) images +This repository contains the top level code that aggregates the various [OpenEmbedded](http://openembedded.org) layers into a whole from which [webOS Open Source Edition (OSE)](https://www.webosose.org/) images can be built. -Description ------------ -This repository contains the top level code that aggregates the various [OpenEmbedded](http://openembedded.org) layers into a whole from which webOS OSE images can be built. +## Prerequisites -Cloning -======= -Set up build-webos by cloning its Git repository: +Before you begin, make sure that you prepare the target device and systems that meet the [System Requirements](https://www.webosose.org/docs/guides/setup/system-requirements/). - git clone https://github.com/webosose/build-webos.git +## How to Build a webOS OSE Image -Note: If you populate it by downloading an archive (zip or tar.gz file), then you will get the following error when you run mcf: +To build a webOS OSE image, refer to [Building webOS OSE](https://www.webosose.org/docs/guides/setup/building-webos-ose/) - fatal: Not a git repository (or any parent up to mount parent). - Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYTEM not set). +If you are already familiar with building webOS OSE, check the following quick summary: +``` bash +# Download this repository +$ git clone https://github.com/webosose/build-webos.git +$ cd build-webos +$ git checkout -t origin/ -Prerequisites -============= -Before you can build, you will need some tools. If you try to build without them, bitbake will fail a sanity check and tell you what's missing, but not really how to get the missing pieces. On Ubuntu, you can force all of the missing pieces to be installed by entering: +# Install and configure the build +$ sudo scripts/prerequisites.sh +$ ./mcf -p 0 -b 0 raspberrypi4-64 - $ cd build-webos - $ sudo scripts/prerequisites.sh +# Start to build +$ source oe-init-build-env +$ bitbake webos-image +``` -Also, the bitbake sanity check will issue a warning if you're not running under Ubuntu 14.04 64bit LTS. +> **Note**: See also [Flashing webOS OSE](https://www.webosose.org/docs/guides/setup/flashing-webos-ose/). +## Copyright and License Information -Building -======== -To configure the build for the raspberrypi3 or qemux86 and to fetch the sources: - - For raspberrypi3 - $ ./mcf -p 0 -b 0 raspberrypi3 - - For qemux86 - $ ./mcf -p 0 -b 0 qemux86 - -The `-p 0` and `-b 0` options set the make and bitbake parallelism values to the number of CPU cores found on your computer. - -To kick off a full build of webOS OSE, make sure you have at least 100GB of disk space available and enter the following: - - $ make webos-image - -This may take in the neighborhood of two hours on a multi-core workstation with a fast disk subsystem and lots of memory, or many more hours on a laptop with less memory and slower disks or in a VM. - - -Images -====== -The following images can be built: - -- `webos-image`: The production webOS OSE image. -- `webos-image-devel`: Adds various development tools to `webos-image`, including gdb and strace. See `packagegroup-core-tools-debug` and `packagegroup-core-tools-profile` in `oe-core` and `packagegroup-webos-test` in `meta-webos` for the complete list. - - -Cleaning -======== -To blow away the build artifacts and prepare to do clean build, you can remove the build directory and recreate it by typing: - - $ rm -rf BUILD - $ ./mcf.status - -What this retains are the caches of downloaded source (under `./downloads`) and shared state (under `./sstate-cache`). These caches will save you a tremendous amount of time during development as they facilitate incremental builds, but can cause seemingly inexplicable behavior when corrupted. If you experience strangeness, use the command presented below to remove the shared state of suspicious components. In extreme cases, you may need to remove the entire shared state cache. See [here](https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#shared-state-cache) for more information on it. - - -Building Individual Components -============================== -To build an individual component, enter: - - $ make +Unless otherwise specified, all content, including all source code files and documentation files in this repository are: -To clean a component's build artifacts under BUILD, enter: +Copyright (c) 2008-2024 LG Electronics, Inc. - $ make clean- +All content, including all source code files and documentation files in this repository except otherwise noted are: Licensed under the Apache License, Version 2.0 (the "License"); you may not use this content except in compliance with the License. You may obtain a copy of the License at -To remove the shared state for a component as well as its build artifacts to ensure it gets rebuilt afresh from its source, enter: +http://www.apache.org/licenses/LICENSE-2.0 - $ make cleanall- +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -Adding new layers -================= -The script automates the process of adding new OE layers to the build environment. The information required for integrate new layer are; layer name, OE priority, repository, identification in the form branch, commit or tag ids. It is also possible to reference a layer from local storage area. The details are documented in weboslayers.py. +SPDX-License-Identifier: Apache-2.0 -Copyright and License Information -================================= -Unless otherwise specified, all content, including all source code files and documentation files in this repository are: +## How to Download Source Codes and Licenses -Copyright (c) 2008-2018 LG Electronics, Inc. +If you *ONLY* want to check the source codes and those license information, enter the following commands: -All content, including all source code files and documentation files in this repository except otherwise noted are: Licensed under the Apache License, Version 2.0 (the "License"); you may not use this content except in compliance with the License. You may obtain a copy of the License at +``` bash +# Download this repository +$ git clone https://github.com/webosose/build-webos.git +$ cd build-webos +$ git checkout -t origin/ -http://www.apache.org/licenses/LICENSE-2.0 +# Install and configure the build +$ sudo scripts/prerequisites.sh +$ ./mcf -p 0 -b 0 raspberrypi4-64 -Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +# Download source codes and licenses +$ source oe-init-build-env +$ bitbake --runall=patch webos-image +``` -SPDX-License-Identifier: Apache-2.0 +You can check the source codes and licenses under the `BUILD/work` directory. diff --git a/build-templates/local-conf.in b/build-templates/local-conf.in index 6194fc39..e0872ceb 100644 --- a/build-templates/local-conf.in +++ b/build-templates/local-conf.in @@ -1,7 +1,7 @@ # DO NOT MODIFY! This script is generated by configure. Changes made # here will be lost. Source for this file is in local-conf.in. -# Copyright (c) 2008-2017 LG Electronics, Inc. +# Copyright (c) 2008-2021 LG Electronics, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -153,7 +153,7 @@ PACKAGE_CLASSES ?= "package_ipk" # Supported values are i686 and x86_64 # # Using ??= so that it can be overridden in webos-local.conf without having to -# resort to using _forcevariable. +# resort to using :forcevariable. SDKMACHINE ??= "${BUILD_ARCH}" # @@ -184,12 +184,9 @@ EXTRA_IMAGE_FEATURES ?= "" # are: # - 'buildhistory' collect statistics from build artifacts # - 'buildstats' collect build statistics -# - 'image-mklibs' to reduce shared library files size for an image # - 'image-prelink' in order to prelink the filesystem image # - 'image-swab' to perform host system intrusion detection -# NOTE: if listing mklibs & prelink both, then make sure mklibs is before prelink -# NOTE: mklibs also needs to be explicitly enabled for a given image, see local.conf.extended -USER_CLASSES ?= "@buildhistory_class@ buildstats image-mklibs" +USER_CLASSES ?= "@buildhistory_class@ buildstats" BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory" BUILDHISTORY_COMMIT ?= "@buildhistory_enabled@" @@ -268,7 +265,7 @@ BB_GENERATE_MIRROR_TARBALLS := "@generatemirrortarballs@" # CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to # track the version of this file when it was generated. This can safely be ignored if # this doesn't mean anything to you. -CONF_VERSION = "1" +CONF_VERSION = "2" # local.conf.sample.extended starts here. # BBMASK is a regular expression that can be used to tell BitBake to ignore @@ -357,15 +354,6 @@ CONF_VERSION = "1" # the toolchain you want to use is included in an appropriate layer # TCMODE = "external-eabi" -# mklibs library size optimization is more useful to smaller images, -# and less useful for bigger images. Also mklibs library optimization -# can break the ABI compatibility, so should not be applied to the -# images which are to be extended or upgraded later. -#This enabled mklibs library size optimization just for the specified image. -#MKLIBS_OPTIMIZED_IMAGES ?= "core-image-minimal" -#This enable mklibs library size optimization will be for all the images. -#MKLIBS_OPTIMIZED_IMAGES ?= "all" - # Uncomment this if your host distribution provides the help2man tool. #ASSUME_PROVIDED += "help2man-native" @@ -412,14 +400,14 @@ CONF_VERSION = "1" # product. If shipped as part of an image these packages may have # implications so they are disabled by default. To enable them, # un-comment the below as appropriate. -#LICENSE_FLAGS_WHITELIST = "commercial_gst-fluendo-mp3 \ -# commercial_gst-openmax \ -# commercial_gst-plugins-ugly \ -# commercial_lame \ -# commercial_libmad \ -# commercial_libomxil \ -# commercial_mpeg2dec \ -# commercial_qmmp" +#LICENSE_FLAGS_ACCEPTED = "commercial_gst-fluendo-mp3 \ +# commercial_gst-openmax \ +# commercial_gst-plugins-ugly \ +# commercial_lame \ +# commercial_libmad \ +# commercial_libomxil \ +# commercial_mpeg2dec \ +# commercial_qmmp" # @@ -433,7 +421,7 @@ CONF_VERSION = "1" # "action,directory,minimum_space,minimum_free_inode" # # The "action" must be set and should be one of: -# ABORT: Immediately abort +# HALT: Immediately halt # STOPTASKS: The new tasks can't be executed any more, will stop the build # when the running tasks have been done. # WARN: show warnings (see BB_DISKMON_WARNINTERVAL for more information) diff --git a/build-templates/oe-init-build-env.in b/build-templates/oe-init-build-env.in index 4724c9ae..db04d1be 100644 --- a/build-templates/oe-init-build-env.in +++ b/build-templates/oe-init-build-env.in @@ -36,7 +36,7 @@ fi # used in runqemu bitbake wrapper for pseudodone location export BUILDDIR="${TOPDIR}/BUILD" -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD" +export BB_ENV_PASSTHROUGH_ADDITIONS="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD" [ -z "${MACHINES}" ] && MACHINES="@machines@" [ -z "${MACHINE}" ] && MACHINE="@machine@" diff --git a/mcf b/mcf index 66b6e4da..00b36cca 100755 --- a/mcf +++ b/mcf @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2008-2017 LG Electronics, Inc. +# Copyright (c) 2008-2024 LG Electronics, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -25,7 +25,7 @@ from time import gmtime, strftime, sleep import shutil import glob -__version__ = "6.2.3" +__version__ = "6.2.9" logger = logging.getLogger(__name__) @@ -52,7 +52,7 @@ def enable_trace(): TRACE = True def enable_clean(): - logger.warn('Running in clean non-interactive mode, all possible local changes and untracked files will be removed') + logger.warning('Running in clean non-interactive mode, all possible local changes and untracked files will be removed') global CLEAN CLEAN = True @@ -247,16 +247,16 @@ def _icecc_installed(): if '1.0.1-1' == iceversion: return True else: - logger.warn("WARNING: Wrong icecc package version {} is installed, disabling build from using ICECC.\n".format(iceversion) + \ + logger.warning("WARNING: Wrong icecc package version {} is installed, disabling build from using ICECC.\n".format(iceversion) + \ "Please check 'How To Install ICECC on Your Workstation (Client)'\n" + \ "http://wiki.lgsvl.com/pages/viewpage.action?pageId=96175316") return False else: - logger.warn('WARNING: ICECC package installation check failed, disabling build from using ICECC.') + logger.warning('WARNING: ICECC package installation check failed, disabling build from using ICECC.') return False except: - logger.warn('WARNING: ICECC package installation check failed, disabling build from using ICECC.') + logger.warning('WARNING: ICECC package installation check failed, disabling build from using ICECC.') return False def location_to_dirname(location): @@ -323,9 +323,18 @@ def wait_for_git_mirror(newcommitid): echo_check_call(cmd) elif not contains_ref(newcommitid): echo_check_call('git remote update && git fetch %s --tags' % REMOTE) + # If the commitid is still not contained even after performing remote update, + # check remote open hash via ls-remote. + if not contains_ref(newcommitid): + refs_change = echo_check_call("git ls-remote %s | grep ^%s | awk '{print $2}'" % (REMOTE, newcommitid)) + if refs_change != "": + echo_check_call("git fetch %s %s >&2" % (REMOTE, refs_change)) + else: + logger.info("MCF-%s Cannot checkout %s in %s" % (__version__, newcommitid, repodir)) + raise Exception('Failed to fetch %s repo' % repodir) success = True break - except subprocess.CalledProcessError: + except (subprocess.CalledProcessError, Exception): sleep(30) if not success: logger.error("MCF-%s Cannot checkout %s in %s" % (__version__, newcommitid, repodir)) @@ -333,7 +342,13 @@ def wait_for_git_mirror(newcommitid): def downloadrepo(layer): cmd = 'git clone %s %s' % (layer["url"], layer["location"]) - echo_check_call(cmd) + for retry in range(5): + try: + echo_check_call(cmd) + break + except subprocess.CalledProcessError: + logger.error("Can't %s retry: %d" % (cmd, retry+1)) + sleep(3) olddir = os.getcwd() os.chdir(layer["location"]) @@ -371,7 +386,10 @@ def downloadrepo(layer): else: # we don't have any branch preference use detached cmd ='git checkout FETCH_HEAD' - echo_check_call(cmd) + try: + echo_check_call(cmd) + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) else: if not contains_ref(newcommitid): wait_for_git_mirror(newcommitid) @@ -384,7 +402,10 @@ def downloadrepo(layer): else: # we don't have any branch preference use detached cmd ='git checkout %s' % newcommitid - echo_check_call(cmd) + try: + echo_check_call(cmd) + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) newtag = layer["tag_new"] if newtag: @@ -396,7 +417,10 @@ def downloadrepo(layer): cmd ='git reset --hard %s' % newtag else: cmd ='git checkout %s' % newtag - echo_check_call(cmd) + try: + echo_check_call(cmd) + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) os.chdir(olddir) @@ -451,7 +475,7 @@ def write_bblayers_conf(sourcedir): locations += "%s_LAYER ?= \"%s/%s\"\n" % (layer_name, topdir, layer["collection_path"]) bblayers += " ${%s_LAYER} \\\n" % layer_name - priorities += "BBFILE_PRIORITY_%s_forcevariable = \"%s\"\n" % (layer["collection_name"], layer["priority"]) + priorities += "BBFILE_PRIORITY_%s:forcevariable = \"%s\"\n" % (layer["collection_name"], layer["priority"]) with open(os.path.join(sourcedir, "conf", "bblayers.conf"), 'a') as f: f.write('\n') @@ -509,7 +533,7 @@ def printupdatesummary (): if not found: logger.info('No local changes found') -def get_remote_branch(newbranch, second_call = False): +def get_remote_branch(newbranch, layer, second_call = False): remotebranch = None refbranchlist = echo_check_call("git branch -r") refbranch = refbranchlist.splitlines() @@ -521,13 +545,18 @@ def get_remote_branch(newbranch, second_call = False): return remotebranch else: # try it again after "git remote update" - echo_check_call("git remote update") - return get_remote_branch(newbranch, True) + try: + echo_check_call("git remote update") + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) + return get_remote_branch(newbranch, layer, True) def reposanitycheck(layer): olddir = os.getcwd() os.chdir(layer["location"]) + logger.info('Sanity checking [%s]' % layer["location"]) + layer["branch_current"] = echo_check_call("git rev-parse --abbrev-ref HEAD").rstrip() res = False @@ -535,7 +564,7 @@ def reposanitycheck(layer): if CLEAN: if echo_check_call("git status --porcelain -s"): layer["sanity_uncommitted_clean"] = True - logger.warn('Removing all local changes and untracked files in [%s]' % layer["location"]) + logger.warning('Removing all local changes and untracked files in [%s]' % layer["location"]) # abort rebase if git pull --rebase from update_layers got stuck on some local commit try: echo_check_call("git rebase --abort 2>/dev/null") @@ -543,19 +572,22 @@ def reposanitycheck(layer): # we can ignore this one pass - echo_check_call("git stash clear") - echo_check_call("git clean -fdx") - echo_check_call("git reset --hard") + try: + echo_check_call("git stash clear") + echo_check_call("git clean -fdx") + echo_check_call("git reset --hard") + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) else: logger.info('Checking for local changes in [%s]' % layer["location"]) if echo_check_call("git status --porcelain --u=no -s"): - logger.warn('Found local uncommitted changes in [%s]' % layer["location"]) + logger.warning('Found local uncommitted changes in [%s]' % layer["location"]) layer["sanity_uncommitted_changes"] = True echo_check_call("git stash") res = True if echo_check_call("git status --porcelain -s | grep -v '^?? MCF-PATCHES_' || true"): - logger.warn('Found local untracked changes in [%s]' % layer["location"]) + logger.warning('Found local untracked changes in [%s]' % layer["location"]) layer["sanity_untracked_changes"] = True res = True @@ -580,13 +612,13 @@ def reposanitycheck(layer): logger.debug("reposanitycheck(%s) dir %s, branchinfo %s, branchinfonew %s, url %s, urlnew %s" % (layer["name"], layer["location"], layer["branch_current"], layer["branch_new"], layer["url"], urlcurrent)) if urlcurrent != layer["url"]: - logger.warn("Changing url for remote '%s' from '%s' to '%s'" % (REMOTE, urlcurrent, layer["url"])) + logger.warning("Changing url for remote '%s' from '%s' to '%s'" % (REMOTE, urlcurrent, layer["url"])) echo_check_call("git remote set-url %s %s" % (REMOTE, layer["url"])) # Sync with new remote repo try: echo_check_call('git remote update') except subprocess.CalledProcessError: - raise Exception('Failed to fetch %s repo' % LOCATIONS[layer]) + raise Exception('Failed to fetch %s repo' % layer["location"]) newbranch = layer["branch_new"] if newbranch: @@ -603,7 +635,7 @@ def reposanitycheck(layer): needcheckout = False break - remotebranch = get_remote_branch(newbranch) + remotebranch = get_remote_branch(newbranch, layer) if foundlocalbranch and remotebranch: if needcheckout: @@ -642,7 +674,7 @@ def reposanitycheck(layer): trackingremote = trackingremote.strip() if not trackingbranch or not trackingremote or trackingbranch.replace('refs/heads',trackingremote) != remotebranch: - logger.warn("checkout %s was tracking '%s/%s' changing it to track '%s'" % (layer["location"], trackingremote, trackingbranch, remotebranch)) + logger.warning("checkout %s was tracking '%s/%s' changing it to track '%s'" % (layer["location"], trackingremote, trackingbranch, remotebranch)) # to ensure we are tracking remote echo_check_call('git branch %s --set-upstream-to %s' % (newbranch, remotebranch)) @@ -650,9 +682,9 @@ def reposanitycheck(layer): echo_check_call('git checkout -b %s %s' % (newbranch, remotebranch)) else: # anything else is failure - raise Exception('Could not find local and remote branches for %s' % newbranch) + raise Exception('Could not find local and remote branch "%s" for layer "%s" from "%s"' % (newbranch, layer["location"], layer["url"])) else: - raise Exception('Undefined branch name') + raise Exception('Undefined branch name for layer "%s" from "%s"' % (layer["location"], layer["url"])) newdir = os.chdir(olddir) return res @@ -692,7 +724,10 @@ def updaterepo(layer): else: # we don't have any branch preference use detached cmd ='git checkout FETCH_HEAD' - echo_check_call(cmd) + try: + echo_check_call(cmd) + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) else: if not contains_ref(newcommitid): wait_for_git_mirror(newcommitid) @@ -705,14 +740,25 @@ def updaterepo(layer): else: # we don't have any branch preference use detached cmd ='git checkout %s' % newcommitid - echo_check_call(cmd) + try: + echo_check_call(cmd) + except subprocess.CalledProcessError: + raise Exception('Failed to fetch %s repo' % layer["location"]) else: - if CLEAN: - echo_check_call("git remote update") - echo_check_call('git reset --hard %s/%s' % (REMOTE, newbranch)) - else: - # current branch always tracks a remote one - echo_check_call('git pull %s' % REMOTE) + for retry in range(5): + try: + if CLEAN: + echo_check_call("git remote update") + echo_check_call('git reset --hard %s/%s' % (REMOTE, newbranch)) + break + else: + # current branch always tracks a remote one + echo_check_call('git pull %s' % REMOTE) + break + except subprocess.CalledProcessError: + logger.error("Can't remote update. retry: %d" % (retry+1)) + if retry+1 == 5: raise Exception('Failed to fetch %s repo' % layer["location"]) + sleep(3) logger.info('Done updating [%s]' % layer["location"]) else: logger.info(('[%s] is up-to-date.' % layer["location"])) @@ -781,21 +827,21 @@ def checkmirror(name, url): if url.startswith('file://'): pathstr = url[7:] if not os.path.isdir(pathstr): - logger.warn("%s parameter '%s' points to non-existent directory" % (name, url)) + logger.warning("%s parameter '%s' points to non-existent directory" % (name, url)) elif not os.listdir(pathstr): - logger.warn("%s parameter '%s' points to empty directory, did you forgot to mount it?" % (name, url)) + logger.warning("%s parameter '%s' points to empty directory, did you forgot to mount it?" % (name, url)) elif len(url) <= 7: logger.error("%s parameter '%s' is incorrect, we expect at least 7 characters for protocol" % (name, url)) def sanitycheck(options): try: - mirror = echo_check_call('git config -l | grep "^url\..*insteadof=github.com/"') + mirror = echo_check_call(r'git config -l | grep "^url\..*insteadof=github.com/"') except subprocess.CalledProcessError: # git config returns 1 when the option isn't set mirror = '' pass if not mirror: - logger.warn('No mirror for github.com was detected, please define mirrors in ~/.gitconfig if some are available') + logger.warning('No mirror for github.com was detected, please define mirrors in ~/.gitconfig if some are available') if options.sstatemirror: for m in options.sstatemirror: if not m: @@ -847,9 +893,9 @@ def configure_build(srcdir, options): ['@icecc_disable_enable@', '1' if not icestate or not options.enable_icecc else ''], ['@icecc_parallel_make@', '%s' % options.icecc_parallel_make], ['@alternative_icecc_installation@', ('ICECC_PATH ?= "%s"' % options.icecc_location) if options.icecc_location else ''], - ['@icecc_user_package_blacklist@', ('ICECC_USER_PACKAGE_BL ?= "%s"' % ' '.join(options.icecc_user_package_blacklist)) if options.icecc_user_package_blacklist else ''], - ['@icecc_user_class_blacklist@', ('ICECC_USER_CLASS_BL ?= "%s"' % ' '.join(options.icecc_user_class_blacklist)) if options.icecc_user_class_blacklist else ''], - ['@icecc_user_package_whitelist@', ('ICECC_USER_PACKAGE_WL ?= "%s"' % ' '.join(options.icecc_user_package_whitelist)) if options.icecc_user_package_whitelist else ''], + ['@icecc_user_package_blacklist@', ('ICECC_RECIPE_DISABLE ?= "%s"' % ' '.join(options.icecc_user_package_blacklist)) if options.icecc_user_package_blacklist else ''], + ['@icecc_user_class_blacklist@', ('ICECC_CLASS_DISABLE ?= "%s"' % ' '.join(options.icecc_user_class_blacklist)) if options.icecc_user_class_blacklist else ''], + ['@icecc_user_package_whitelist@', ('ICECC_RECIPE_ENABLE ?= "%s"' % ' '.join(options.icecc_user_package_whitelist)) if options.icecc_user_package_whitelist else ''], ['@icecc_environment_script@', 'ICECC_ENV_EXEC ?= "%s"' % options.icecc_env_exec if options.icecc_location else ''], ['@icecc_disable_enable_mcf@', '--%s-icecc' % ('disable' if not icestate or not options.enable_icecc else 'enable')], ['@alternative_icecc_installation_mcf@', options.icecc_location if options.icecc_location else ''], diff --git a/oe-logging.json b/oe-logging.json new file mode 100644 index 00000000..4b9085c1 --- /dev/null +++ b/oe-logging.json @@ -0,0 +1,23 @@ +{ + "version": 1, + "formatters": { + "warnlogFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + "handlers": { + "warnlog": { + "class": "logging.FileHandler", + "formatter": "warnlogFormatter", + "level": "WARNING", + "filename": "warn.log" + } + }, + "loggers": { + "BitBake": { + "handlers": ["warnlog"] + } + }, + "@disable_existing_loggers": false +} diff --git a/scripts/build.sh b/scripts/build.sh index 3464daef..6916a82a 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -1,6 +1,6 @@ #!/bin/bash -# Copyright (c) 2013-2017 LG Electronics, Inc. +# Copyright (c) 2013-2024 LG Electronics, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -18,7 +18,7 @@ #set -x # Some constants -SCRIPT_VERSION="6.10.5a" +SCRIPT_VERSION="6.11.10" SCRIPT_NAME=`basename $0` AUTHORITATIVE_OFFICIAL_BUILD_SITE="rpt" @@ -79,8 +79,20 @@ function print_timestamp { printf "TIME: ${SCRIPT_NAME}-${SCRIPT_VERSION} $1: ${BUILD_TIMESTAMP}, +${BUILD_TIMEDIFF}, +${BUILD_TIMEDIFF_START}, ${BUILD_TIMESTAMPH}\n" | tee -a ${BUILD_TIME_LOG} } +function check_bitbake_processes { + DIR=$1 + if ps aux | grep ..*bitbake | grep ${DIR}; then + echo "ERROR: There are some bitbake processes already running in ${DIR} directory, refusing to start another build here" + last -n 30 + ps aux | grep bitbake + exit 1 + fi +} + print_timestamp "start" +check_bitbake_processes ${BUILD_TOPDIR} + declare -i RESULT=0 function showusage { @@ -91,6 +103,7 @@ OPTIONS: -T, --targets Targets to build (unlike images they aren't copied from buildhistory) -M, --machines Machines to build -b, --bom Generate BOM files + -o, --only-bom Generate webOS BOM only without image -s, --signatures Dump sstate signatures, useful to compare why something is rebuilding -u, --scp-url scp will use this path to download and update \${URL}/latest_project_baselines.txt and also @@ -101,6 +114,7 @@ OPTIONS: -B, --buildhistory-ref Branch where to push buildhistory for repositories on gerrit it should start with refs/heads replaces detecting it from JOB_NAME and JENKINS_URL + -C, --codename Codename, replaces detecting it from JOB_NAME -V, --version Show script version -h, --help Print this help message ! @@ -201,16 +215,68 @@ function check_project_vars { echo "$ldesc" } +function unset_buildhistory_commit { + [ -f webos-local.conf ] && sed -i '/BUILDHISTORY_COMMIT/d' webos-local.conf + echo "BUILDHISTORY_COMMIT = \"0\"" >> webos-local.conf +} + +function set_buildhistory_commit { + if [ -n "${BUILD_BUILDHISTORY_PUSH_REF}" ]; then + [ -f webos-local.conf ] && sed -i '/BUILDHISTORY_COMMIT/d' webos-local.conf + echo "BUILDHISTORY_COMMIT = \"1\"" >> webos-local.conf + fi +} + +function copy_webos_bom { + MACHINE=$1 + I=$2 + F=$3 + BASE_BOM=`git describe --always --match "builds/${BUILD_CODENAME}/*" | awk -F"-" '{print $1}' | awk -F"/" '{print $NF}'` + if ( echo "${BASE_BOM}" | grep -qE "^[0-9]+$" ); then + [ -d ${ARTIFACTS}/${MACHINE}/${I} ] || mkdir -p ${ARTIFACTS}/${MACHINE}/${I} + echo "copy_webos_bom" + + wget "http://webosimg.lge.com/images/${DISTRO}-${BUILD_CODENAME}-official-${MACHINE}/${BASE_BOM}/${MACHINE}/${I}/webos-bom.json" -O "${ARTIFACTS}/${MACHINE}/${I}/${F}" + if [ $? -ne 0 ]; then + echo "Failed to download ${F}" + return 1 + fi + wget "http://webosimg.lge.com/images/${DISTRO}-${BUILD_CODENAME}-official-${MACHINE}/${BASE_BOM}/${MACHINE}/${I}/webos-bom-full.json" -O "${ARTIFACTS}/${MACHINE}/${I}/webos-bom-full-before.json" + wget "http://webosimg.lge.com/images/${DISTRO}-${BUILD_CODENAME}-official-${MACHINE}/${BASE_BOM}/${MACHINE}/${I}/webos-compile-option.json" -O "${ARTIFACTS}/${MACHINE}/${I}/webos-compile-option-before.json" + wget "http://webosimg.lge.com/images/${DISTRO}-${BUILD_CODENAME}-official-${MACHINE}/${BASE_BOM}/${MACHINE}/${I}/webos-common-compile-option.json" -O "${ARTIFACTS}/${MACHINE}/${I}/webos-common-compile-option-before.json" + ls "${ARTIFACTS}/${MACHINE}/${I}" + else + return 1 + fi +} + function generate_webos_bom { MACHINE=$1 I=$2 F=$3 rm -f webos-bom.json + rm -f webos-bom-sort.json + rm -f webos-bom-full.json + rm -f webos-bom-full-sort.json + rm -f webos-compile-option.json + rm -f webos-compile-option-sort.json + rm -f webos-common-compile-option.json + unset_buildhistory_commit /usr/bin/time -f "$TIME_STR" bitbake --runall=write_bom_data ${I} 2>&1 | tee /dev/stderr | grep '^TIME:' >> ${BUILD_TIME_LOG} [ -d ${ARTIFACTS}/${MACHINE}/${I} ] || mkdir -p ${ARTIFACTS}/${MACHINE}/${I} - sort webos-bom.json > ${ARTIFACTS}/${MACHINE}/${I}/${F} - rm -f webos-bom.json + sort webos-bom.json > webos-bom-sort.json + sort webos-bom-full.json > webos-bom-full-sort.json + sort webos-compile-option.json > webos-compile-option-sort.json + if [[ "${F}" =~ "before" ]]; then + COMPILE_SUFFIX="-before" + elif [[ "${F}" =~ "after" ]]; then + COMPILE_SUFFIX="-after" + fi + sed -e '1s/^{/[{/' -e '$s/,$/]/' webos-compile-option-sort.json > ${ARTIFACTS}/${MACHINE}/${I}/webos-compile-option${COMPILE_SUFFIX}.json + cp webos-common-compile-option.json ${ARTIFACTS}/${MACHINE}/${I}/webos-common-compile-option${COMPILE_SUFFIX}.json + sed -e '1s/^{/[{/' -e '$s/,$/]/' webos-bom-full-sort.json > ${ARTIFACTS}/${MACHINE}/${I}/webos-bom-full${COMPILE_SUFFIX}.json + sed -e '1s/^{/[{/' -e '$s/,$/]/' webos-bom-sort.json > ${ARTIFACTS}/${MACHINE}/${I}/${F} } function filter_images { @@ -225,6 +291,7 @@ function filter_images { function call_bitbake { filter_images + set_buildhistory_commit /usr/bin/time -f "$TIME_STR" bitbake ${BBFLAGS} ${FILTERED_IMAGES} ${TARGETS} 2>&1 | tee /dev/stderr | grep '^TIME:' >> ${BUILD_TIME_LOG} # Be aware that non-zero exit code from bitbake doesn't always mean that images weren't created. @@ -250,8 +317,8 @@ function add_md5sums_and_buildhistory_artifacts { ${ARTIFACTS}/${MACHINE}/${I}/*.tar.bz2 \ ${ARTIFACTS}/${MACHINE}/${I}/*.rpi-sdimg \ ${ARTIFACTS}/${MACHINE}/${I}/*.rpi-sdimg.gz \ - ${ARTIFACTS}/${MACHINE}/${I}/*.wic \ - ${ARTIFACTS}/${MACHINE}/${I}/*.wic.gz \ + ${ARTIFACTS}/${MACHINE}/${I}/*.wic.bmap \ + ${ARTIFACTS}/${MACHINE}/${I}/*.wic.bz2 \ ${ARTIFACTS}/${MACHINE}/${I}/*.zip \ ${ARTIFACTS}/${MACHINE}/${I}/*.vfat \ ${ARTIFACTS}/${MACHINE}/*.fastboot \ @@ -286,13 +353,15 @@ function add_buildhistory_artifacts { BHMACHINE=`echo ${MACHINE} | sed 's/-/_/g'` if ls buildhistory/sdk/*/${I} >/dev/null 2>/dev/null; then if [ -n "${SDKMACHINE}" ] ; then - for d in buildhistory/sdk/${I}-${SDKMACHINE}-*; do - [ -d ${ARTIFACTS}/${SDKMACHINE}/${MACHINE} ] || mkdir -p ${ARTIFACTS}/${SDKMACHINE}/${MACHINE}/ - cp -a $d/${I} ${ARTIFACTS}/${SDKMACHINE}/${MACHINE}/ + for MACHINE in ${BMACHINES}; do + for d in buildhistory/sdk/${DISTRO}-sdk-${I}-${SDKMACHINE}-*-${MACHINE}; do + [ -d ${ARTIFACTS}/${SDKMACHINE}/${MACHINE} ] || mkdir -p ${ARTIFACTS}/${SDKMACHINE}/${MACHINE} + cp -a $d/${I} ${ARTIFACTS}/${SDKMACHINE}/${MACHINE}/ + done done else - for d in buildhistory/sdk/*; do - mkdir -p ${ARTIFACTS}/${MACHINE}/ + for d in buildhistory/sdk/${DISTRO}-sdk-${I}-*-${MACHINE}; do + [ -d ${ARTIFACTS}/${MACHINE} ] || mkdir -p ${ARTIFACTS}/${MACHINE}/ cp -a $d/${I} ${ARTIFACTS}/${MACHINE}/ done fi @@ -310,15 +379,21 @@ function add_buildhistory_artifacts { # manager.addInfoBadge("${manager.build.getWorkspace().child('BUILD-ARTIFACTS/build-id.txt').readToString()}") # in all builds (making BUILD_IMAGES/BUILD_MACHINE changes less error-prone) FIRST_IMAGE="${MACHINE}/${I}" - ln -vnf ${ARTIFACTS}/${BHMACHINE}/${I}/build-id.txt ${ARTIFACTS}/build-id.txt + ln -vnf ${ARTIFACTS}/${MACHINE}/${I}/build-id.txt ${ARTIFACTS}/build-id.txt fi ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/image-info.txt ${ARTIFACTS}/${MACHINE}/${I}/image-info.txt - ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/files-in-image.txt ${ARTIFACTS}/${MACHINE}/${I}/files-in-image.txt + ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/files-in-image* ${ARTIFACTS}/${MACHINE}/${I}/ ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/installed-packages.txt ${ARTIFACTS}/${MACHINE}/${I}/installed-packages.txt + ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/installed-package-names.txt ${ARTIFACTS}/${MACHINE}/${I}/installed-package-names.txt ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/installed-package-sizes.txt ${ARTIFACTS}/${MACHINE}/${I}/installed-package-sizes.txt if [ -e buildhistory/images/${BHMACHINE}/glibc/${I}/installed-package-file-sizes.txt ] ; then ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/installed-package-file-sizes.txt ${ARTIFACTS}/${MACHINE}/${I}/installed-package-file-sizes.txt fi + if [ -f buildhistory/images/${BHMACHINE}/glibc/${I}/ls2_api_list.json ]; then + ln -vn buildhistory/images/${BHMACHINE}/glibc/${I}/ls2_api_list.json ${ARTIFACTS}/${MACHINE}/${I}/ls2_api_list.json + else + echo "Warning: There is no ls2_api_list.json at 'buildhistory/images/${BHMACHINE}/glibc/${I}'" + fi fi } @@ -344,20 +419,27 @@ function move_kernel_image_and_add_symlinks { } function move_artifacts { + # XXX Might there be other subdirectories under BUILD/deploy that weren't created by this build? + # Some MACHINEs like raspberrypi4-64 now contain dash which gets converted to underscore for MACHINE_ARCH + BHMACHINE=`echo ${MACHINE} | sed 's/-/_/g'` for I in ${FILTERED_IMAGES}; do mkdir -p "${ARTIFACTS}/${MACHINE}/${I}" || true # we store only tar.gz, vmdk.zip and .epk images # and we don't publish kernel images anymore - if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk >/dev/null 2>/dev/null; then - if type zip >/dev/null 2>/dev/null; then - # zip vmdk images if they exists - find BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk -exec zip -j {}.zip {} \; - ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk.zip ${ARTIFACTS}/${MACHINE}/${I}/ - else - # report failure and publish vmdk - echo "ERROR: ${SCRIPT_NAME}-${SCRIPT_VERSION} zip utility isn't installed on the build server" >&2 - RESULT+=1 - ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk ${ARTIFACTS}/${MACHINE}/${I}/ + if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk* >/dev/null 2>/dev/null; then + if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk >/dev/null 2>/dev/null; then + if type zip >/dev/null 2>/dev/null; then + # zip vmdk images if they exists + find BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk -exec zip -j {}.zip {} \; + ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk.zip ${ARTIFACTS}/${MACHINE}/${I}/ + else + # report failure and publish vmdk + echo "ERROR: ${SCRIPT_NAME}-${SCRIPT_VERSION} zip utility isn't installed on the build server" >&2 + RESULT+=1 + ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk ${ARTIFACTS}/${MACHINE}/${I}/ + fi + elif ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk.gz >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vmdk.gz ${ARTIFACTS}/${MACHINE}/${I}/ fi # copy webosvbox if we've built vmdk image if [ -e meta-webosose/meta-webos/scripts/webosvbox -a ! -e ${ARTIFACTS}/${MACHINE}/webosvbox ] ; then @@ -403,13 +485,16 @@ function move_artifacts { if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vfat >/dev/null 2>/dev/null; then ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.vfat ${ARTIFACTS}/${MACHINE}/${I}/ fi - elif ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic >/dev/null 2>/dev/null; then - if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic >/dev/null 2>/dev/null; then - gzip -f BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic - ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.gz ${ARTIFACTS}/${MACHINE}/${I}/ + elif ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.bz2 >/dev/null 2>/dev/null; then + if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.bz2 >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.bz2 ${ARTIFACTS}/${MACHINE}/${I}/ + fi + if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.bmap >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.wic.bmap ${ARTIFACTS}/${MACHINE}/${I}/ fi if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 >/dev/null 2>/dev/null; then ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 ${ARTIFACTS}/${MACHINE}/${I}/ + rm -vf ${ARTIFACTS}/${MACHINE}/${I}/${I}-${MACHINE}-*.ostree.tar.bz2 fi elif ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.gz >/dev/null 2>/dev/null \ || ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.fastboot >/dev/null 2>/dev/null \ @@ -431,7 +516,7 @@ function move_artifacts { if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.epk >/dev/null 2>/dev/null; then ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.epk ${ARTIFACTS}/${MACHINE}/${I}/ fi - elif BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 >/dev/null 2>/dev/null \ + elif ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 >/dev/null 2>/dev/null \ || ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.zip >/dev/null 2>/dev/null; then if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 >/dev/null 2>/dev/null; then ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.tar.bz2 ${ARTIFACTS}/${MACHINE}/${I}/ @@ -439,17 +524,61 @@ function move_artifacts { if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.zip >/dev/null 2>/dev/null; then ln -vn BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.zip ${ARTIFACTS}/${MACHINE}/${I}/ fi - elif ls BUILD/deploy/sdk/${I}-*.sh >/dev/null 2>/dev/null; then + elif ls BUILD/deploy/sdk/${DISTRO}-sdk-${I}-*-${MACHINE}-*.sh >/dev/null 2>/dev/null; then if [ -n "${SDKMACHINE}" ] ; then - [ -d ${ARTIFACTS}/${SDKMACHINE} ] || mkdir -p ${ARTIFACTS}/${SDKMACHINE} - ln -vn BUILD/deploy/sdk/${I}-*.sh ${ARTIFACTS}/${SDKMACHINE}/ + for MACHINE in ${BMACHINES}; do + local DEST_DIR=${ARTIFACTS}/${SDKMACHINE}/${MACHINE}/${I}/ + [ -d ${DEST_DIR} ] || mkdir -p ${DEST_DIR} + for ext in sh host.manifest target.manifest testdata.json; do + ln -vn BUILD/deploy/sdk/${DISTRO}-sdk-${I}-${SDKMACHINE}-*-${MACHINE}-*.${ext} ${DEST_DIR} + done + rmdir --ignore-fail-on-non-empty ${ARTIFACTS}/${MACHINE}/${I} ${ARTIFACTS}/${MACHINE} + done else - ln -vn BUILD/deploy/sdk/${I}-*.sh ${ARTIFACTS}/${MACHINE}/${I}/ + local DEST_DIR=${ARTIFACTS}/${MACHINE}/${I}/ + [ -d ${DEST_DIR} ] || mkdir -p ${DEST_DIR} + for ext in sh host.manifest target.manifest testdata.json; do + ln -vn BUILD/deploy/sdk/${DISTRO}-sdk-${I}-*-${MACHINE}-*.${ext} ${DEST_DIR} + done fi else echo "WARN: No ${I} images with recognized IMAGE_FSTYPES found to copy as build artifacts" fi + if ls BUILD/deploy/images/${MACHINE}/${I}-oss-pkg-info.yaml >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-oss-pkg-info.yaml ${ARTIFACTS}/${MACHINE}/${I}/oss-pkg-info.yaml + else + echo "WARN: No oss-pkg-info.yaml to copy as build artifacts" + fi + + if ls BUILD/deploy/images/${MACHINE}/${I}-dependency.json >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-dependency.json ${ARTIFACTS}/${MACHINE}/${I}/dependency.json + else + echo "WARN: No dependency.json to copy as build artifacts" + fi + + if ls BUILD/deploy/images/${MACHINE}/${I}-cve.json >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-cve.json ${ARTIFACTS}/${MACHINE}/${I}/cve.json + else + echo "WARN: No cve.json to copy as build artifacts" + fi + + if ls BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.spdx.* >/dev/null 2>/dev/null; then + for SPDX in BUILD/deploy/images/${MACHINE}/${I}-${MACHINE}-*.spdx.*; do + if echo $SPDX | grep -q "${WEBOS_DISTRO_BUILD_ID}"; then + ln -vn $(readlink -f ${SPDX}) ${ARTIFACTS}/${MACHINE}/${I}/ + fi + done + else + echo "WARN: No spdx files to copy as build artifacts" + fi + + if ls BUILD/deploy/images/${MACHINE}/${I}-checksec.json >/dev/null 2>/dev/null; then + ln -vn BUILD/deploy/images/${MACHINE}/${I}-checksec.json ${ARTIFACTS}/${MACHINE}/${I}/checksec.json + else + echo "WARN: No checksec.json to copy as build artifacts" + fi + # delete possibly empty directories rmdir --ignore-fail-on-non-empty ${ARTIFACTS}/${MACHINE}/${I} ${ARTIFACTS}/${MACHINE} done @@ -468,11 +597,14 @@ function move_artifacts { fi fi + [ -f warn.log ] && ln -vn warn.log ${ARTIFACTS}/warn.log + [ -f ccache_stats.json ] && ln -vn ccache_stats.json ${ARTIFACTS}/ccache_stats.json + move_kernel_image_and_add_symlinks add_md5sums_and_buildhistory_artifacts } -TEMP=`getopt -o I:T:M:S:j:J:B:u:bshV --long images:,targets:,machines:,scp-url:,site:,jenkins:,job:,buildhistory-ref:,bom,signatures,help,version \ +TEMP=`getopt -o I:T:M:S:j:J:B:u:C:bsohV --long images:,targets:,machines:,scp-url:,site:,jenkins:,job:,buildhistory-ref:,codename:,bom,only-bom,signatures,help,version \ -n $(basename $0) -- "$@"` if [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 2 ; fi @@ -490,7 +622,9 @@ while true ; do -J|--job) BUILD_JOB="$2" ; shift 2 ;; -B|--buildhistory-ref) BUILD_BUILDHISTORY_PUSH_REF="$2" ; shift 2 ;; -u|--scp-url) URL="$2" ; shift 2 ;; + -C|--codename) BUILD_CODENAME="$2" ; shift 2 ;; -b|--bom) CREATE_BOM="Y" ; shift ;; + -o|--only-bom) CREATE_BOM="O" ; shift ;; -s|--signatures) SIGNATURES="Y" ; shift ;; -h|--help) showusage ; shift ;; -V|--version) echo ${SCRIPT_NAME} ${SCRIPT_VERSION}; exit ;; @@ -551,7 +685,7 @@ fi # Generate BOM files with metadata checked out by mcf (pinned versions) if [ -n "${CREATE_BOM}" -a -n "${BMACHINES}" ]; then print_timestamp "before first bom" - if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integ" -o "${BUILD_JOB}" = "engr" -o "${BUILD_JOB}" = "clean" ] ; then + if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integr" -o "${BUILD_JOB}" = "engr" -o "${BUILD_JOB}" = "clean" ] ; then # don't use -before suffix for official builds, because they don't need -after and .diff because # there is no logic for using different revisions than weboslayers.py BOM_FILE_SUFFIX="-before" @@ -560,6 +694,10 @@ if [ -n "${CREATE_BOM}" -a -n "${BMACHINES}" ]; then for MACHINE in ${BMACHINES}; do filter_images for I in ${FILTERED_IMAGES} ${TARGETS}; do + if [ "${BOM_FILE_SUFFIX}" = "-before" ] ; then + copy_webos_bom "${MACHINE}" "${I}" "webos-bom${BOM_FILE_SUFFIX}.json" && continue + echo "copy_webos_bom from webosimg.lge.com failed" + fi generate_webos_bom "${MACHINE}" "${I}" "webos-bom${BOM_FILE_SUFFIX}.json" done done @@ -567,7 +705,7 @@ fi print_timestamp "before verf/engr/clean logic" -if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integ" -o "${BUILD_JOB}" = "engr" ] ; then +if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integr" -o "${BUILD_JOB}" = "engr" ] ; then if [ "$GERRIT_PROJECT" != "${BUILD_REPO}" ] ; then set -e # checkout issues are critical for verification and engineering builds for project in "${BUILD_LAYERS[@]}" ; do @@ -592,7 +730,7 @@ fi # Generate BOM files again, this time with metadata possibly different for engineering and verification builds if [ -n "${CREATE_BOM}" -a -n "${BMACHINES}" ]; then - if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integ" -o "${BUILD_JOB}" = "engr" -o "${BUILD_JOB}" = "clean" ] ; then + if [ "${BUILD_JOB}" = "verf" -o "${BUILD_JOB}" = "mlverf" -o "${BUILD_JOB}" = "integr" -o "${BUILD_JOB}" = "engr" -o "${BUILD_JOB}" = "clean" ] ; then print_timestamp "before 2nd bom" . oe-init-build-env for MACHINE in ${BMACHINES}; do @@ -602,6 +740,9 @@ if [ -n "${CREATE_BOM}" -a -n "${BMACHINES}" ]; then diff ${ARTIFACTS}/${MACHINE}/${I}/webos-bom-before.json \ ${ARTIFACTS}/${MACHINE}/${I}/webos-bom-after.json \ > ${ARTIFACTS}/${MACHINE}/${I}/webos-bom-diff.txt + diff ${ARTIFACTS}/${MACHINE}/${I}/webos-compile-option-before.json \ + ${ARTIFACTS}/${MACHINE}/${I}/webos-compile-option-after.json \ + > ${ARTIFACTS}/${MACHINE}/${I}/webos-compile-option-diff.txt done done fi @@ -610,6 +751,8 @@ fi print_timestamp "before signatures" if [ -n "${SIGNATURES}" -a -n "${BMACHINES}" ]; then + unset_buildhistory_commit + . oe-init-build-env for MACHINE in ${BMACHINES}; do mkdir -p "${ARTIFACTS}/${MACHINE}" || true @@ -640,12 +783,25 @@ else echo "INFO: buildhistory won't be pushed because buildhistory directory isn't git repo or BUILD_BUILDHISTORY_PUSH_REF wasn't set" fi +sed '/^BB_LOGCONFIG = "${TOPDIR}.oe-logging.json"/d' -i webos-local.conf +if [ -f ${BUILD_TOPDIR}/oe-logging.json ] ; then + echo "INFO: enabling logger config from ${BUILD_TOPDIR}/oe-logging.json" + echo "BB_LOGCONFIG = \"\${TOPDIR}/oe-logging.json\"" >> webos-local.conf +fi + print_timestamp "before main '${JOB_NAME}' build" FIRST_IMAGE= if [ -z "${BMACHINES}" ]; then echo "ERROR: calling build.sh without -M parameter" +elif [ "${CREATE_BOM}" = "O" -a -n "${BMACHINES}" ]; then + echo "[INFO] Generate webOS BOM only" + cd "${CALLDIR}" + exit 0; else + # remove old ccache_stats.json + rm -fv ccache_stats.json + . oe-init-build-env if [ -n "${BUILD_SDKMACHINES}" ] && echo "${IMAGES}" | grep -q "^[^- ]\+-bdk$" ; then # if there is only one image ending with "-bdk" and BUILD_SDKMACHINES is defined diff --git a/scripts/prerequisites.sh b/scripts/prerequisites.sh index e1257709..0200cf66 100755 --- a/scripts/prerequisites.sh +++ b/scripts/prerequisites.sh @@ -1,6 +1,6 @@ #!/bin/bash -e -# Copyright (c) 2008-2018 LG Electronics, Inc. +# Copyright (c) 2008-2024 LG Electronics, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,11 +14,11 @@ # See the License for the specific language governing permissions and # limitations under the License. -# This has only been tested on Ubuntu-12.04 and Ubuntu-14.04 amd64. +# This has only been tested on Ubuntu-16.04, Ubuntu-18.04 and Ubuntu-20.04 amd64. check_sanity=true usage="$0 [--help|-h] [--version|-V] [--force|-f]" -version="2.2.3" +version="2.3.0" statusfile="/etc/webos.prerequisites" for i ; do @@ -62,10 +62,13 @@ fi sane=true distributor_id_sane="^Ubuntu$" -release_sane="^14.04$|^16.04$|^18.04$" -codename_sane="^trusty$|^xenial$|^bionic$" +release_sane="^20.04$|^22.04$" +codename_sane="^focal$|^jammy$" arch_sane="^amd64$" +# Below packages shouled be installed before this script runs. +apt-get install software-properties-common lsb-release -y + case "${check_sanity}" in true) if [ ! -x /usr/bin/lsb_release ] ; then @@ -118,25 +121,21 @@ esac # locales, because utf8 is needed with newer bitbake which uses python3 essential="\ build-essential \ + python3-distutils \ chrpath \ cpio \ diffstat \ + file \ gawk \ git \ iputils-ping \ + liblz4-tool \ locales \ - lsb-release \ - python \ - python2.7 \ python3 \ - texinfo \ wget \ + zstd \ " -# add python3-distutils only for 18.04 (and later should be added for newer) -# because it doesn't exist as separate package in older Ubuntu releases -[ `/usr/bin/lsb_release -s -r` = "18.04" ] && essential="${essential} python3-distutils" - # bzip2, gzip, tar, zip are used by our scripts/build.sh archivers="\ bzip2 \ @@ -147,19 +146,13 @@ archivers="\ # gcc-multilib is needed to build 32bit version of pseudo # g++-multilib is needed to build and run 32bit mksnapshot of V8 (in chromium53) +# time is used by scripts/build.sh extras="\ gcc-multilib \ g++-multilib \ time \ " -# add ppa for openjdk -java_version=`java -version 2>&1 | head -n 1 | cut -d\" -f 2 | cut -d\. -f 1` -if [ -z $java_version ] || [ $java_version -lt 10 ]; then - add-apt-repository ppa:openjdk-r/ppa - extras="${extras} openjdk-11-jdk" -fi - apt-get update apt-get install --yes \ @@ -167,10 +160,6 @@ apt-get install --yes \ ${extras} \ ${archivers} \ -if [ -z $java_version ] || [ $java_version -lt 10 ]; then - update-alternatives --set java /usr/lib/jvm/java-11-openjdk-amd64/bin/java -fi - locale-gen en_US.utf8 echo $version > $statusfile diff --git a/weboslayers.py b/weboslayers.py index 17ed97b4..3c4967d0 100644 --- a/weboslayers.py +++ b/weboslayers.py @@ -1,4 +1,4 @@ -# Copyright (c) 2008-2019 LG Electronics, Inc. +# Copyright (c) 2008-2024 LG Electronics, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -53,33 +53,34 @@ Distribution = "webos" # Supported MACHINE-s -Machines = ['qemux86', 'raspberrypi3', 'raspberrypi4'] +Machines = ['qemux86', 'qemux86-64', 'qemuarm', 'raspberrypi3', 'raspberrypi3-64', 'raspberrypi4', 'raspberrypi4-64'] # github.com/openembedded repositories are read-only mirrors of the authoritative # repositories on git.openembedded.org webos_layers = [ -('bitbake', -1, 'git://github.com/openembedded/bitbake.git', 'branch=1.40,commit=8a58c3c6', ''), +('bitbake', -1, 'https://github.com/openembedded/bitbake.git', 'branch=2.8,commit=aa0e540fc31', ''), -('meta', 5, 'git://github.com/openembedded/openembedded-core.git', 'branch=thud,commit=d3d3f44303', 'oe-core'), +('meta', 5, 'https://github.com/openembedded/openembedded-core.git', 'branch=scarthgap,commit=37835788d07', 'oe-core'), -('meta-oe', 10, 'git://github.com/openembedded/meta-openembedded.git', 'branch=thud,commit=2d088d2526', 'meta-oe'), -('meta-multimedia', 11, 'git://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), -('meta-networking', 12, 'git://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), -('meta-python', 13, 'git://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), -('meta-filesystems', 14, 'git://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), +('meta-oe', 10, 'https://github.com/openembedded/meta-openembedded.git', 'branch=scarthgap,commit=dda0d533260', 'meta-oe'), +('meta-multimedia', 11, 'https://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), +('meta-networking', 12, 'https://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), +('meta-python', 13, 'https://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), +('meta-filesystems', 14, 'https://github.com/openembedded/meta-openembedded.git', '', 'meta-oe'), -('meta-updater', 15, 'git://github.com/advancedtelematic/meta-updater.git', 'branch=thud,commit=ee1548d', ''), -('meta-qt5', 20, 'git://github.com/meta-qt5/meta-qt5.git', 'branch=warrior,commit=dc2dedef', ''), +('meta-clang', 15, 'https://github.com/kraj/meta-clang.git', 'branch=scarthgap,commit=8c77b427408', ''), +('meta-virtualization', 16, 'https://git.yoctoproject.org/git/meta-virtualization', 'branch=scarthgap,commit=2ec66285d05', ''), -('meta-webos-backports-2.7', 30, 'git://github.com/webosose/meta-webosose.git', '', ''), -('meta-webos-backports-2.8', 31, 'git://github.com/webosose/meta-webosose.git', '', ''), +('meta-qt6', 20, 'https://code.qt.io/yocto/meta-qt6.git', 'branch=6.8.1,commit=v6.8.1', ''), -('meta-webos', 40, 'git://github.com/webosose/meta-webosose.git', 'branch=master,commit=92eef16', ''), -('meta-webos-updater', 41, 'git://github.com/webosose/meta-webosose.git', '', ''), +('meta-webos-backports-5.1', 34, 'https://github.com/webosose/meta-webosose.git', '', ''), +('meta-webos-backports-5.2', 35, 'https://github.com/webosose/meta-webosose.git', '', ''), -('meta-raspberrypi', 50, 'git://git.yoctoproject.org/meta-raspberrypi', 'branch=thud,commit=89bdf97', ''), -('meta-webos-raspberrypi', 51, 'git://github.com/webosose/meta-webosose.git', '', ''), -('meta-webos-smack', 75, 'git://github.com/webosose/meta-webosose.git', '', ''), +('meta-webos', 40, 'https://github.com/webosose/meta-webosose.git', 'branch=master,commit=d7ed46c33c5', ''), -('meta-security', 77, 'git://git.yoctoproject.org/meta-security', 'branch=thud,commit=31dc4e7', ''), +('meta-raspberrypi', 50, 'https://github.com/agherzan/meta-raspberrypi.git', 'branch=scarthgap,commit=6df7e028a2', ''), +('meta-webos-raspberrypi', 51, 'https://github.com/webosose/meta-webosose.git', '', ''), +('meta-webos-virtualization', 53, 'https://github.com/webosose/meta-webosose.git', '', ''), + +('meta-security', 76, 'https://git.yoctoproject.org/git/meta-security', 'branch=scarthgap,commit=bc865c5276c', ''), ]