Yulij Andreevich Lesov
- node stack
- rails stack
- php stack
- data stores
- search indexing
- version control
Hong Kong Polytechnic University
University smartcard access control system
Hong Kong Polytechnic University had been using a smartcard-based access control system for almost 10 years. The original system was built on Windows XP; however, with its support officially terminating in 2014, management was concerned about the future stability and security of the system. A new system built upon Windows 7 - the most recent Windows version at the time - was commissioned.
The new system has, above all, compatibility with all existing behavior and smooth transition as its goal. The system will be deployed at multiple sites across Hong Kong Polytechnic University, each site having slightly different access control rules; but all share usage of the same smartcard.
A node-express stack was chosen to build the core control logic and user interface display. This architecture choice is to leverage the speed of the V8 engine, and to allow building a more user friendly and intuitive interface.
Core access control logic is built in node: having access to a high level language make implementing the differing access control rules relatively trivial.
A browser-based user interface is chosen, for its being the most ubiquitous and familiar interface available, thus enabling a no-training-required transition to the new system. It is also the least intrusive method to enable full use of the underlying Windows system, which end users would need for, e.g. reporting done via Microsoft Office.
The display is also designed and fine-tuned specifically for the actual use cases. On one hand, it maximizes available space for displaying information. On the other hand, it uses a large enough font size, plus sharp color contrast, to cater for security guards' ease of use.
Actual display design parameters chosen are based upon iterative user feedback from field testing, ensuring actual usability from actual users of the system.
The smartcard is a Mifare card, and its low-level read-write layer is written in C#. This enables directly utilizing the APIs of the Windows driver for the card reader. The C# layer, then, interacts with the main access control logic in node, via calling APIs exposed from the node side.
Being a Windows-based system, batch files are written for scheduled jobs. This include system autostart, user list fetching ,and regular backup routines.
The solution also includes features for organizational governance use. Auditing is available via raw log exports; reporting is available via formatted log exports, which are then edited in Microsoft Office. Backups are performed both regularly, and on-demand through the interface. All these transparently conform to the expected behaviors and formatting of the existing system, again having the smooth transition goal in mind.
Finally, security is paramount on any access control system. All data on the smartcard are encrypted; only card readers with a dedicated decryption key will be able to perform meaningful read/write on the cards. The deployed machines have full network access blocked - including intranet, except for a single whitelisted access to the user list server. The user list server itself is accessible only within intranet, and even within this, only accessible by whitelisted machines.
Kicks For Pennies
A live penny-auction platform built for traffic spikes
Kicks For Pennies started its penny auction platform from an off-the-shelf PHP-based solution, which validated its business idea and served the first batch of customers. As it has found its foothold in the market, the original platform started to become sluggish and unpredictable upon heavier traffic.
The business nature of a penny auction platform has very concentrated peak traffic, all during the final few sections of an auction. A solution that can reliably serve this peak traffic is critically needed for the continued viability of Kicks For Pennies. While addressing this issue, Kicks For Pennies has also decided to introduce a membership scheme to build brand loyalty.
A new platform is built on rails, leveraging its speed of development and maturity of packages.
The key part - real time auction bidding - is separated out and built on its own architecture. Redis is used as real time bidding data store, for speed, instead of the application's main persistent data store (mysql). Websockets is chosen as communication channel, for its lighter overhead, instead of the main http protocol. Finally, barebones node + express is used as glue code to bring the pieces together in this architecture.
Sidekiq jobs are deployed for background data processing. It acts as audit double check against real time bidding, ensuring data consistency: accurate auction states and customer account balances. Jobs are also deployed to streamline the business, mainly through a semi-automatic auction lister based on pre-defined heuristical rules.
Sidekiq jobs are also used in third party integrations, syncing with inventory management solution, delivering user notifications, generating physical labels and documents for logistics, etc.
From a growing startup to a maturing business, this iteration also includes more systematic business management solutions: integrated audit trail, platform-specific analytics, regular key data reports.
Android and iOS mobile frontends through phonegap are also developed, trading earlier launch for speed and features.
Subscription-based e-course platform with corporate website
LRN provides corporate compliance education services, and seeks to build an e-platform to host its resources in e-version. It also runs a parallel aggressive rebranding focus with a new corporate website.
The platform runs a subscription-based business model, with accounts being managed in-platform. It hosts courses, files, webinars, and videos, fronted by an online catalog. Materials are fully indexed and searchable, with an added feature that generates recommendations based on the account's profile.
The platform has also integrated with an existing legacy system, due to its being already in use widely in production.
The platfrorm runs an API-client architecture: backend runs on node (express); frontend angular.
Node-based architecture gives horizontal scalability to the backend, by use of multiple cores with clusters. The wide array of npm packages fit the needs of the backend, which has to be more general-purpose than a typical API server. Tasks range from background-processing uploaded videos, handling Excel and PDF files, pre-processing resources for indexing, etc.
Another aspect is integrating with multiple third party service providers, e.g. marketing system, analytics software, video hosting, etc. These provide general developer APIs, but with added integration modules for popular platforms - node being one.
The corporate website is developed separately like an individual project, with also a node-angular architecture.
It features a set of custom-designed page templates, plus matching content widgets. This setup combines and achieves both the refinement and coherence from having custom, and polished designs; while retaining the flexibility for user-initiated edits on dynamic content.
It is also fully responsive, adapting to mobile, tablet, desktop, and HD screens.
Search and indexing module for fashion e-commerce platform
ThredUP is an e-commerce platform selling second hand brandwear. This gives a far greater than typical variety of inventory; while its users expect an interface no less user-friendly than any other fashion e-commerce platform.
A dedicated module to optimize search and indexing is needed to serve a user-friendly frontend. This new module is developed alongside the originally quick-hack solution built upon rails + sunspot. It is being continually tested, refined, and iterated upon during development, gradually phasing out the original sunspot-based solution; all the while not interrupting existing functionality already in production.
Apache lucene, fronted by solr, is the engine for this search implementation. Shell scripts, sidekiq workers, and rails model callbacks, together handle the data retrieval, transformation, and indexing.
Textual data are parsed both as complete strings and tokenized searchables. The former caters for readily available data display, e.g., on category names; the latter allows implementing full text searches and lookaheads user interfaces.
Discrete categorical data, including textual data, are indexed by the underlying unique IDs; while the corresponding display labels are stored alongside but not indexed. This allows for filtering-based searches while having data available in returned results for display. Boolean data is treated as discrete categorical data with two possible values.
Numerical data - price - is indexed as normalized integers - price being stored in cents. This avoids floating point math inaccuracies in any layer from any possible implementation. All conversions between integer and decimal display are made as close to the user interface layer as possible. For display, this happens only at the template layer; for input, this happens at the value stored in the underlying implementation of the input control.
Color recognition and categorization on items is performed by a dedicated algorithm, operating on pictures of the item. The resulting data is treated as a discrete categorical data.
Data indexing and synchronization happen at two entry points. Push-based indexing happen upon data modification - insert, update, or delete - the data is marked dirty for indexing. Pull-based indexing is implemented by nightly data pulls; this is intended as a safeguard against any data inconsistency.
All data indexing happen with sidekiq workers, to avoid jamming the main app execution thread.
It is also possible to force an immediate reindexing of any specific data point, should the business need arise, through an admin interface.
A "Saved Search" feature is implemented on member accounts, by storing exact search parameters as freeform json in main user database record.
Hong Kong Jockey Club Vending Machines
Embedded system for payment gateway integration on vending machines
Hong Kong Jockey Club wants to modify a fleet of vending machines with Weixin Wallet, an online payment gateway built upon a social network platform. The solution would drive sales to its merchandise by capturing the segment already used to Weixin Wallet, as well as providing marketing opportunities via customers' social profiles.
The vending machines are based upon the MDB protocol, a prevalent standard among vending machines. Payment integration includes an embedded Linux, then a C++ layer accessing the machine's MDB APIs, finally another userland code layer for the actual payment gateway integration.
A cloud rails server serves as the central backend for all machines deployed. It handles the payment gateway integration, exposing an API for embedded machine software to call. Both push- and pull- communications between machines and server are built through socket.io.
The project features very heavy defensive coding, due to the unattended nature of vending machines. All imaginable errors, including any mis-usage from user themselves, must be conservatively handled, with the overall goal being avoiding any sales disputes.
Ansible playbooks have been created for the project, for accessible and predictable deploys on a large fleet of vending machines.
The accompanying control panel is built on node-angular, and deployed within docker containers.
Both the machine server and the control panel server shares the same underlying data source, but each accesses and modifies the data via its own self-contained layer.
The panel allows staff to enter product information, which would be displayed on the Weixin mobile app screens during transaction.
It also features an intuitive presentation of transaction logs, they let staff resolve disputes easily, while retaining all essential information for auditing purposes.
Smart customer service platform with heuristic logic
Tuul is built as an online platform that unifies and digitizes the customer service industry. Customer service centres, call centres, ticket-based workflows: all these are operational practices of customer service operators. Tuul is designed to encompass these workflows under an accessible interface.
The chat is chosen as the main interaction interface, mirroring operators' workflows where customer interaction is cornerstone. From this baseline, features are designed around the chat interface. Meaningful customer interaction related actions are accessed from the chat itself. Related information, such as the case files and histories, are placed next to the chat.
Heuristics suggest commonly used customer service actions based on analyzing chat content. These are implemented as self-contained modules, chat bots.
The frontend is built on angular: snappiness in chat interface is key in this interface design. Other components in the architecture thus follow.
Cassandra is chosen as the main persistent data store. A deployed production cassandra cluster provides an eventual consistency model on each mode, thus enabling fast reads and non-blocking writes. Redis is chosen as a fast in-memory general-purpose data store, available for access from any business logic - and used for components dealing with near-real time features.
This is unified in a node-express layer, which has common API access to both cassandra and redis, while also being friendly to building the platform's own API for frontend consumption.
Eventual developer access to the platform API is also planned for: API endpoints are versioned, and as structured as possible amidst the developing business logic. The chat itself is implemented through XMPP, thus conforming to another standardized protocol, versus a custom in-house one.