CI: try to fix and simplify apt setup

Run apt update to update the package cache, as this seems to be needed by
Travis now. Also reduce the package set - we don't build any deb or rpm
packages anymore. This should hopefully fix the CI build.

.travis.yml

@@ -9,25 +9,10 @@ python:
   - "3.6"
   - "pypy3"
 before_install:
-  - sudo apt install rpm dpkg-dev debhelper dh-python python3-setuptools fakeroot python3-serial python3-yaml
+  - sudo apt update
+  - sudo apt install python3-setuptools python3-serial python3-yaml
 install:
-  - pip install pyserial pyusb tqdm
+  - pip install pyusb
 script:
   - python setup.py build
   - python setup.py test
-before_deploy:
-  - deactivate
-  - python3 setup.py bdist_rpm
-  - dpkg-buildpackage -uc -us
-  - cp ../*.deb dist/
-deploy:
-  provider: releases
-  api_key: $GH_TOKEN
-  file_glob: true
-  file:
-  - dist/stcgal*_all.deb
-  - dist/stcgal*.noarch.rpm
-  skip_cleanup: true
-  on:
-    tags: true
-    python: "3.4"

README.md

@@ -32,6 +32,17 @@ Features
 * Automatic power-cycling with DTR toggle or a custom shell command
 * Automatic UART protocol detection
 
+Quickstart
+----------
+
+Install stcgal (might need root/administrator privileges):
+    
+    pip3 install stcgal
+
+Call stcgal and show usage:
+
+    stcgal -h
+
 Further information
 -------------------
 

debian/changelog

@@ -1,3 +1,9 @@
+stcgal (1.6) unstable; urgency=low
+
+  * Update to 1.6
+
+ -- Grigori Goronzy <greg@chown.ath.cx>  Mon, 24 Sep 2018 22:56:31 +0200
+
 stcgal (1.4) unstable; urgency=low
 
   * Update to 1.4

debian/docs

@@ -1,2 +1,4 @@
 README.md
-TODO.md
+doc/USAGE.md
+doc/MODELS.md
+doc/FAQ.md

doc/FAQ.md

@@ -7,7 +7,7 @@ By design, this is not possible with STC's bootloader protocols. This is conside
 
 ### Which serial interfaces have been tested with stcgal?
 
-The following USB-based UART interface chips have been successfully tested with stcgal:
+stcgal should work fine with common 16550 compatible UARTs that are traditionally available on many platforms. However, nowadays, USB-based UARTs are the typical case. The following USB-based UART interface chips have been successfully tested with stcgal:
 
 * FT232 family (OS: Linux, Windows)
 * CH340/CH341 (OS: Windows, Linux requires Kernel 4.10)
@@ -20,6 +20,10 @@ Interfaces that are known to not work:
 
 In general, stcgal requires accurate baud rate timings and parity support.
 
+### stcgal fails to start with the error `module 'serial' has no attribute 'PARITY_NONE'` or similar
+
+There is a module name conflict between the PyPI package 'serial' (a data serialization library) and the PyPI package 'pyserial' (the serial port access library needed by stcgal). You have to uninstall the 'serial' package (`pip3 uninstall serial`) and reinstall 'pyserial' (`pip3 install --force-reinstall pyserial`) to fix this. There is no other known solution at the moment.
+
 ### stcgal fails to recognize the MCU and is stuck at "Waiting for MCU"
 
 There are a number of issues that can result in this symptom:
@@ -40,7 +44,7 @@ Various remedies are possible to avoid parasitic powering.
 
 First, make sure that the frequency specified uses the correct unit. The frequency is specified in kHz and the safe range is approximately 5000 kHz - 30000 kHz. Furthermore, frequency trimming uses the UART clock as the clock reference, so UART incompatibilities or clock inaccuracies can also result in issues with frequency trimming. If possible, try another UART chip.
 
-### Baud rate switching fails
+### Baud rate switching fails or flash programming fails
 
 This can especially happen at high programming baud rates, e.g. 115200 baud. Try a lower baudrate, or stick to the default of 19200 baud. Some USB UARTs are known to cause problems due to inaccurate timing as well, which can lead to various issues.
 

doc/INSTALL.md

@@ -1,10 +1,17 @@
 Installation
 ============
 
-stcgal requires Python 3.2 (or later) and pySerial. USB support is
+stcgal requires Python 3.2 (or later), pyserial 3.0 or later and
-optional and requires pyusb 1.0.0b2 or later. You can run stcgal
+TQDM 4.0.0 or later. USB support is optional and requires pyusb
-directly with the included ```stcgal.py``` script. The recommended
+1.0.0b2 or later. You can run stcgal directly with the included
-method for permanent installation is to use Python's setuptools. Run
+```stcgal.py``` script if the dependencies are already installed.
-```./setup.py build``` to build and ```sudo ./setup.py install```
+
-to install stcgal. A permanent installation provides the ```stcgal```
+There are several options for permanent installation:
-command.
+
+* Use Python3 and ```pip```. Run ```pip3 install stcgal``` to
+install the latest release of stcgal globally on your system.
+This may require administrator/root permissions for write access
+to system directories.
+
+* Use setuptools. Run ```./setup.py build``` to build and
+```sudo ./setup.py install``` to install stcgal.

doc/MODELS.md

@@ -1,7 +1,7 @@
 Supported MCU models
 ====================
 
-stcgal should fully support STC 89/90/10/11/12/15 series MCUs. Support for STC8 series MCUs is work in progress.
+stcgal should fully support STC 89/90/10/11/12/15/8 series MCUs.
 
 So far, stcgal was tested with the following MCU models:
 

doc/PyPI.md

@@ -0,0 +1,18 @@
+stcgal - STC MCU ISP flash tool
+===============================
+
+stcgal is a command line flash programming tool for [STC MCU Ltd](http://stcmcu.com/).
+8051 compatible microcontrollers.
+
+STC microcontrollers have an UART/USB based boot strap loader (BSL). It
+utilizes a packet-based protocol to flash the code memory and IAP
+memory over a serial link. This is referred to as in-system programming
+(ISP).  The BSL is also used to configure various (fuse-like) device
+options. Unfortunately, this protocol is not publicly documented and
+STC only provide a (crude) Windows GUI application for programming.
+
+stcgal is a full-featured Open Source replacement for STC's Windows
+software; it supports a wide range of MCUs, it is very portable and
+suitable for automation.
+
+[See the GitHub page for more information](https://github.com/grigorig/stcgal).

doc/USAGE.md

@@ -4,12 +4,14 @@ Usage
 Call stcgal with ```-h``` for usage information.
 
 ```
-usage: stcgal.py [-h] [-a] [-P {stc89,stc12a,stc12,stc15a,stc15,auto}]
+usage: stcgal.py [-h] [-a] [-r RESETCMD]
+                 [-P {stc89,stc12a,stc12b,stc12,stc15a,stc15,stc8,usb15,auto}]
                  [-p PORT] [-b BAUD] [-l HANDSHAKE] [-o OPTION] [-t TRIM] [-D]
+                 [-V]
                  [code_image] [eeprom_image]
 
-stcgal 1.0 - an STC MCU ISP flash tool
+stcgal 1.5 - an STC MCU ISP flash tool
-(C) 2014-2015 Grigori Goronzy
+(C) 2014-2018 Grigori Goronzy and others
 https://github.com/grigorig/stcgal
 
 positional arguments:
@@ -20,18 +22,20 @@ optional arguments:
   -h, --help            show this help message and exit
   -a, --autoreset       cycle power automatically by asserting DTR
   -r RESETCMD, --resetcmd RESETCMD
-                        Use this shell command for board power-cycling
+                        shell command for board power-cycling (instead of DTR
-                        (instead of DTR assertion)
+                        assertion)
-  -P {stc89,stc12a,stc12,stc15a,stc15,auto}, --protocol {stc89,stc12a,stc12,stc15a,stc15,auto}
+  -P {stc89,stc12a,stc12b,stc12,stc15a,stc15,stc8,usb15,auto}, --protocol {stc89,stc12a,stc12b,stc12,stc15a,stc15,stc8,usb15,auto}
-                        protocol version
+                        protocol version (default: auto)
   -p PORT, --port PORT  serial port device
   -b BAUD, --baud BAUD  transfer baud rate (default: 19200)
   -l HANDSHAKE, --handshake HANDSHAKE
                         handshake baud rate (default: 2400)
   -o OPTION, --option OPTION
-                        set option (can be used multiple times)
+                        set option (can be used multiple times, see
-  -t TRIM, --trim TRIM  RC oscillator frequency in kHz (STC15 series only)
+                        documentation)
+  -t TRIM, --trim TRIM  RC oscillator frequency in kHz (STC15+ series only)
   -D, --debug           enable debug output
+  -V, --version         print version info and exit
 ```
 
 Most importantly, ```-p``` sets the serial port to be used for programming.
@@ -43,6 +47,7 @@ BSL. The protocol can be specified with the ```-P``` flag. By default
 UART protocol autodetection is used. The mapping between protocols
 and MCU series is as follows:
 
+* ```auto``` Automatic detection of UART based protocols (default)
 * ```stc89``` STC89/90 series
 * ```stc12a``` STC12x052 series and possibly others
 * ```stc12b``` STC12x52 series, STC12x56 series and possibly others
@@ -51,11 +56,10 @@ and MCU series is as follows:
 * ```stc15``` Most STC15 series
 * ```stc8``` STC8 series
 * ```usb15``` USB support on STC15W4 series
-* ```auto``` Automatic detection of UART based protocols (default)
 
-The text files in the doc/ subdirectory provide an overview over
+The text files in the doc/reverse-engineering subdirectory provide an
-the reverse engineered protocols used by the BSLs. For more details,
+overview over the reverse engineered protocols used by the BSLs. For
-please read the source code.
+more details, please read the source code.
 
 ### Getting MCU information
 
@@ -92,6 +96,8 @@ Target options:
 Disconnected!
 ```
 
+If the identification fails, see the [FAQ](FAQ.md) for troubleshooting.
+
 ### Program the flash memory
 
 stcgal supports Intel HEX encoded files as well as binary files. Intel
@@ -195,8 +201,8 @@ If the internal RC oscillator is used (```clock_source=internal```),
 stcgal can execute a trim procedure to adjust it to a given value. This
 is only supported by STC15 series and newer. The trim values are stored
 with device options. Use the ```-t``` flag to request trimming to a certain
-value. Generally, frequencies between 4 and 35 MHz can be achieved. If
+value. Generally, frequencies between 4000 and 30000 kHz can be achieved.
-trimming fails, stcgal will abort.
+If trimming fails, stcgal will abort.
 
 ### Automatic power-cycling
 

setup.py

@@ -24,14 +24,14 @@
 import stcgal
 from setuptools import setup, find_packages
 
-with open("README.md", "r") as fh:
+with open("doc/PyPI.md", "r") as fh:
     long_description = fh.read()
 
 setup(
     name = "stcgal",
     version = stcgal.__version__,
     packages = find_packages(exclude=["doc", "tests"]),
-    install_requires = ["pyserial"],
+    install_requires = ["pyserial>=3.0", "tqdm>=4.0.0"],
     extras_require = {
         "usb": ["pyusb>=1.0.0"]
     },
@@ -50,7 +50,7 @@ setup(
     license = "MIT License",
     platforms = "any",
     classifiers = [
-        "Development Status :: 4 - Beta",
+        "Development Status :: 5 - Production/Stable",
         "Environment :: Console",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: MIT License",

stcgal/__init__.py

@@ -1 +1 @@
-__version__ = "1.4"
+__version__ = "1.6"

stcgal/frontend.py

@@ -147,6 +147,10 @@ class StcGal:
     def run(self):
         """Run programmer, main entry point."""
 
+        if self.opts.version:
+            print("stcgal {}".format(stcgal.__version__))
+            return 0
+
         try:
             self.protocol.connect(autoreset=self.opts.autoreset, resetcmd=self.opts.resetcmd)
             if isinstance(self.protocol, StcAutoProtocol):
@@ -178,6 +182,10 @@ class StcGal:
             sys.stdout.flush()
             print("I/O error: %s" % ex, file=sys.stderr)
             return 1
+        except Exception as ex:
+            sys.stdout.flush()
+            print("Unexpected error: %s" % ex, file=sys.stderr)
+            return 1
 
         try:
             if self.opts.code_image:
@@ -214,19 +222,20 @@ def cli():
     # check arguments
     parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,
                                      description="stcgal {} - an STC MCU ISP flash tool\n".format(stcgal.__version__) +
-                                                 "(C) 2014-2017 Grigori Goronzy\nhttps://github.com/grigorig/stcgal")
+                                                 "(C) 2014-2018 Grigori Goronzy and others\nhttps://github.com/grigorig/stcgal")
     parser.add_argument("code_image", help="code segment file to flash (BIN/HEX)", type=argparse.FileType("rb"), nargs='?')
     parser.add_argument("eeprom_image", help="eeprom segment file to flash (BIN/HEX)", type=argparse.FileType("rb"), nargs='?')
     parser.add_argument("-a", "--autoreset", help="cycle power automatically by asserting DTR", action="store_true")
-    parser.add_argument("-r", "--resetcmd",  help="Use this shell command for board power-cycling (instead of DTR assertion)", action="store")
+    parser.add_argument("-r", "--resetcmd",  help="shell command for board power-cycling (instead of DTR assertion)", action="store")
     parser.add_argument("-P", "--protocol", help="protocol version (default: auto)",
                         choices=["stc89", "stc12a", "stc12b", "stc12", "stc15a", "stc15", "stc8", "usb15", "auto"], default="auto")
     parser.add_argument("-p", "--port", help="serial port device", default="/dev/ttyUSB0")
     parser.add_argument("-b", "--baud", help="transfer baud rate (default: 19200)", type=BaudType(), default=19200)
     parser.add_argument("-l", "--handshake", help="handshake baud rate (default: 2400)", type=BaudType(), default=2400)
-    parser.add_argument("-o", "--option", help="set option (can be used multiple times)", action="append")
+    parser.add_argument("-o", "--option", help="set option (can be used multiple times, see documentation)", action="append")
-    parser.add_argument("-t", "--trim", help="RC oscillator frequency in kHz (STC15 series only)", type=float, default=0.0)
+    parser.add_argument("-t", "--trim", help="RC oscillator frequency in kHz (STC15+ series only)", type=float, default=0.0)
     parser.add_argument("-D", "--debug", help="enable debug output", action="store_true")
+    parser.add_argument("-V", "--version", help="print version info and exit", action="store_true")
     opts = parser.parse_args()
 
     # run programmer

tests/stc8f2k08s2-untrimmed.yml

@@ -0,0 +1,5 @@
+name: STC8F2K08S2 untrimmed programming test
+protocol: stc8
+code_data: [49, 50, 51, 52, 53, 54, 55, 56, 57]
+responses:
+  - [0x50, 0xFF, 0xFF, 0xFF, 0xFF, 0x8F, 0x00, 0x04, 0xFF, 0xFF, 0x8B, 0xFD, 0xFF, 0x27, 0x38, 0xF5, 0x73, 0x73, 0x55, 0x00, 0xF6, 0x41, 0x0A, 0x88, 0x86, 0x6F, 0x8F, 0x08, 0x20, 0x20, 0x20, 0x01, 0x00, 0x00, 0x20, 0x05, 0x3C, 0x18, 0x05, 0x22, 0x32, 0xFF]

tests/test_program.py

@@ -27,6 +27,7 @@ from unittest.mock import patch
 import yaml
 import stcgal.frontend
 import stcgal.protocols
+from stcgal.protocols import StcProtocolException
 
 def convert_to_bytes(list_of_lists):
     """Convert lists of integer lists to list of byte lists"""
@@ -43,6 +44,7 @@ def get_default_opts():
     opts.trim = 22118
     opts.eeprom_image = None
     opts.debug = False
+    opts.version = False
     opts.code_image.name = "test.bin"
     opts.code_image.read.return_value = b"123456789"
     return opts
@@ -127,6 +129,24 @@ class ProgramTests(unittest.TestCase):
         """Test a programming cycle with STC15 protocol, L1 series"""
         self._program_yml("./tests/stc15l104w.yml", serial_mock, read_mock)
 
+    @patch("stcgal.protocols.StcBaseProtocol.read_packet")
+    @patch("stcgal.protocols.Stc89Protocol.write_packet")
+    @patch("stcgal.protocols.serial.Serial", autospec=True)
+    @patch("stcgal.protocols.time.sleep")
+    @patch("sys.stdout")
+    def test_program_stc8_untrimmed(self, out, sleep_mock, serial_mock, write_mock, read_mock):
+        """Test error with untrimmed MCU"""
+        with open("./tests/stc8f2k08s2-untrimmed.yml") as test_file:
+            test_data = yaml.load(test_file.read())
+            opts = get_default_opts()
+            opts.trim = 0.0
+            opts.protocol = test_data["protocol"]
+            opts.code_image.read.return_value = bytes(test_data["code_data"])
+            serial_mock.return_value.inWaiting.return_value = 1
+            read_mock.side_effect = convert_to_bytes(test_data["responses"])
+            gal = stcgal.frontend.StcGal(opts)
+            self.assertEqual(gal.run(), 1)
+
     def test_program_stc15w4_usb(self):
         """Test a programming cycle with STC15W4 USB protocol"""
         self.skipTest("USB not supported yet, trace missing")