OS/2 on NT Plan white paper
---------------------------
Author: Yaron Shamir.
The first 6 chapters define goals, assumptions,
constraints, design alternatives and tasks.
The rest break the tasks down and estimate the effort and
experty needed to accomplish the tasks.
Other helpful writeups are:
* .\pownt.ppt (a slide show that summerizes the project)
* .\ioctls.txt (Cruiser IOCTLs support)
* .\apistat (start point of cruiser 32 API implementation)
* \nt\public\spec\os2.txt (original design point of SS)
Note: this paper is still in a working mode i.e. draft. It is
checked in for convenience of reader.
0. Basics:
----------
The overall mission is to construct a SW layer, on top of NT OS,
that will have similar functionality to OS/2 2.0 (base, PM and basic
utilities).
1. Goals:
---------
This are the goals defined for the 'OS2 on NT' effort. an (*) sign means
'reservations, assumptions etc. follow'.
1 'clean' X86 32 bit apps of OS2 2.0 should run unmodified (*)
2 the OS/2 layer should be portable accros HW platforms.
3 'well-behaving' PM 16-bit apps (OS/2 1.1, 1.2, 1.3) will run binary
compatible.
6 A complete set of system services will be provided.
- base APIs (*)
- PM APIs (32 and 16) (*)
- CMD, shell, spooler, task manager, control panel,
file manager will be integrated with WIN32, as much as schedule
allows.
- NLS support (OS2 NLS and DBCS).
2. Assumptions:
---------------
2.1 Image Format:
-----------------
OS/2 2.0 will adopt PE EXE format, such that the NT loader will load
32 bit Cruiser apps. We will provide a 16-bit EXE (NE) loader.
2.2 Alignment:
--------------
OS/2 2.0 will support natural alignment as defined by DCRs.
2.3 Exception Handling:
----------------------
OS2 on NT will support the NT type exception handling.
A DCR was filed for OS2 2.0 to support it as well.
2.4 OS2 2.0 API:
---------------
Is frozen.
3 Constraints and Caveats:
--------------------------
3.1 General Restrictions on apps:
---------------------------------
- No mixed 16/32 code.
- No direct HW manipulation:
* No private Decive Drivers
* Only a limited set of IOCTLS will be supported
(see list in \\popcorn\razzle!src\os2\ioctls.txt)
* No ring-2 code (IOPL code)
* No direct manipulation of physical video buffer i.e. VIO apps
will run inside a character window only.
- No use of unpublished entry points.
3.2 NT features not in the plan for PM:
--------------------------------------
- C2 security.
- prevasive exception handling thru-out the PM code. We intend to
put enough exception handling to make it safe for the rest of
the system.
4 Building OS2 on NT in phases:
------------------------------
Phase Approx MY
----- ---------
0 Build the team 2 (2 month per Engeineer, 10 Eng)
1 32b Base APIs 2 (+1 if LE needed and SEH not accepted)
2 16 bit VIO 5
3 PM, split screens 10
4 'share the glass' 10
4.1 Summary of the phases:
--------------------------
This section summerizes the phases by current applications support,
specific restrictions for the phase, portablility, status,
work items and WIN/NT support. See also ???\pownt.ppt.
Further details are in chapter ???, 'task break down'.
4.1.1 Phase 1 Summary:
----------------------
Features:
* Support Cruiser DOS32 APIs only. They run within the WIN32/NT
environment (CMD, TM, FM).
* Not supported (in addition to general restrictions): PM,
VIO/KBD/MOU/MON, IBM 2.0 exception handling.
* Applications that can run: CRT, 32 bit flat. Examples: SQL, C,
UNIX-ported apps.
* Portablility: Easy. MIPS and X86 can be supported at the same
time.
Status:
* Largely complete, compatible with MS Cruiser
as of 7/90.
Work:
* Loader for LE - Flatten and Widen Cruiser loader.
* Named pipes layer.
* NLS ("Country.sys" etc), map on Unicode.
* Device + FS IOCTLs (limited).
* Misc (complete semaphores, session APIs)
* Integration of OS2 base server with WIN/NT server.
Support from Win/NT:
* Assist loading non-PE executables (if need to
load LE).
* NLS support.
* HPFS support (FM).
4.1.2 Phase 2 Summary:
----------------------
Features:
* Support 16bit OS2 1.X , non-PM (VIO) apps, inside a character
window.
* Not supported (in addition to general restrictions): None.
* Applications that can run: WORD5.5, Paradox, Rbase, PWB.
* Portability: hard. Require emulation for non-x86, alike the work for
WIn16 .
Status:
* No work done so far.
Work:
* Loader for 16b OS2 binaries (NE ).
* Provide a 16-32 thunk layer.
* Implement the following 16B OS2 APIs:
- VIO, MOU, MON, KBD support.
- 16B style semaphores (FSRAM etc.).
- 16B memory management.
- Signal handling.
Support from Win/NT:
* LDT management.
* 16B context switch.
* console support for VIO.
4.1.3 Phase 3 Summary:
----------------------
* 'Frame for PM applications' - not a system of it's own - no
separate PM file manager, print manager, control panel etc. Support
16bit and 32bit clean PM binaries.
* Not supported (in addition to general restrictions): Clipboard, DDE
between PM apps and Windows apps.
* Application that can run: PM-WORD, PM-EXCEL, 123G ...
* Portability: 32 bit easy, 16 bit requires emulation.
Status:
* Win32 engine can support PMGRE, need hooks.
* Most PMWIN is flattened and widened to 32.
Work:
* Write a PMGRE mapping layer to sit on GDI.
* Flatten and widen PM GPI (sits on PMGRE).
* Complete port PMWIN.Hook into WIN32 input.
* 16->32 thunking.
* Integrate with WIN32 - remoting, input etc.
* Unicode to NLS/DBCS.
* Configuration APIs (os2sys.ini, os2.ini).
Support from Win/NT:
* USER/Input: Recongize the 'PM session', SPB.
* GDI: PMWIN callbacks. PM-flavor DC, item variants.
4.1.4 Phase 4 Summary:
----------------------
Features:
* OS2 and Windows app sharing the glass and communicating with
each other.
* Clean PM apps (defined by ???) run unmodified (binary).
* No restrictions except general restrictions in ???.
Status:
Not started.
Work:
* Design necessary hooks in USER and input.
* Implement handshake between PMWIN and USER.
* Implement PM frame windows as USER windows ('a bit of Holeport')
* Implement translation layers to support DDE, clipboard, metafile.
* More environment itegration (e.g. task manager).
Support from Win/NT:
* Recognize PM-flavour windows (the PM 'frame' windows). Pass input
directed to those, to the PM queue.
5 Design notes:
--------------
5.1 Phase 3: Portable, split screen groups:
-------------------------------------------
This is the approach that is recommended by the experts. It calls
for remoting the PM apis selectively from the client DLL into
a PM server that reside with the WIN32 server and uses it.
The Approach - a virtual desktop:
+++++++++++++++++++++++++++++++++
Generaly, the way we see the PM/OS2 subsystem, is that of a desktop, one
of many. It is NOT viewed as a counter part to the NT+WIN32 system, but
rather as a subsystem on top of it. What we mean by this is that for the
user there is one physical desktop, managed in one place. The PM 'virtual
desktop' is just a frame for running PM apps. The implication of this
view are pervasive, e.g. it does not make sense to provide a PM
print manager user interface.
+++++++++++++++++++++++++++++++++
5.1.1 Split Screen block diagram:
--------------------------------
PM app process WIN32 + PM server process
--------------
PM client DLL PMWIN module WIN USER module
-----+-------- ------------ ----------------
| GPI GRE =======> GDI module
| ------------ ----------------
| DDI (display driver)
------------ ------------ ----------------
BASE client DLL OS2 base server WIN32 kernel module
--------------- ---------------------------------------
| |
NT apis + lpc (warp?) lpc + NT APIs
| |
| |
------v------------------------------------|-----------------
+------------------------------------+
NT Executive
-----------------------------------------------------------
The picture is too rough to cover all aspects. The intention is to
build PMWIN as a separate unit that (as a rule) don't use the windows USER
module. The reason for that is mainly the list of hard-to-solve differences
between the two. For example, different order of messages
(which implies a different state machine for windows, hence a different
design) between PMWIN and USER.
This was experienced by porthole project as a major problem in merging the
two window managers. Hence the constraints of different screen groups,
no DDE and no Clipboard between the two.
On the other hand, it is perceived feasible, though non-trivial,
to write a PMGRE layer that maps onto WIN32 GDI graphic engine.
As for display and input drivers, the intention is to share the WIN32
drivers all the way i.e. no PM display device drivers or input device
drivers are needed.
The OS/2 kernel (base) layer is completely independant of the PM/WIndows
issues and sits directly on top of NT, with a client DLL and a server
process DLL, to be merged into the WINDOWS server (csr).
Some extra support (detailed thru the sections below), will be required
from the WIN32 server modules: USER to support the input model and
background/foreground switching, GDI and/or DDI to support PM-style
graphics.
5.1.2 Implementation Notes
--------------------------
5.1.2.1 Base:
------------
We will take the current base OS/2 layer and complete it to cover all
APIs (see constraints for the match to Cruiser). The main items missings
today are:
- named pipes APIs
- session APIs
- NLS DBCS implementation (crosses many APIs).
- portions of MuxWait APIs.
- portions of Memory management APIs.
- portions of EAs APIs (FindFirst/Next/Close)
- 16 bit support.
Character based utilities: CMD32.EXE and it's built-in utilities are done
(80%). The other utilities are written to use WIN32 base calls.
We can offer those to the OS/2 user, since CMD32.EXE will be able to
invoke apps written for other layers, via the NT SM. However - the prefered
approach is to eliminate cmd32.exe all together, and have CMD.EXE be the
only CLI in the system, invoking both flavours of apps. The cmd line
semantics is almost identical between WIN, DOS and OS2 CMDs.
5.1.2.3 PM 'Presentation' Device Drivers:
-----------------------------------------
The term 'presentation DD' refers to the graphic I/O devices that
are driven by the PM graphic engine. Those are NOT based on standard
NT I/O Device Drivers, but are rather a set of user mode programs that
control devices directly. Such are screen, printers, scanners. The
interface they present is called NTDDI and is used EXCLUSIVELY by the
PM engine. This model is NOT used for the user input devices (see section
on PMWIN and input system.
For the presentation (or display) device drivers (and in short DDI), the
PM implementation simply do not exist. Since we use the WIN32 engine,
this part of the system us unified for WIN32 and PM (thanks to the engine
design).
5.1.2.4 PM GPI/GRE:
-------------------
Here is the architecure picture, now zoomed on the graphic engine area:
+----------------+
| PM Application ---------+
+-------|--------+ | Client
. .
--------------------------------------------------------------
. .
v | Server
+-------+ |
| PMWIN + |
+---+---+ v
| +-----------+
| | PM GPI |
| +-----V-----+
+PMGRE API->| PM GRE |
| +-----|-----+
| v
| +----------------------------+
M-Desktop xface+---->| WIN32 GDI Graphic Engine |
+----------------------------+
NT DDI Interface -----------------------------------
+----------------------------+
| Display DD | Printer DD |..|
+----------------------------+
We will implement a PMGRE layer to sit on top of the GDI engine. We will
take the IBM PMGPI layer and complete it (this means porting to 32 bit flat
portable code), using the Cruiser GPI code. This approach will keep the rest
of the system (PMWIN, PM apps ...) untouched.
Engine integration Notes:
-------------------------
- Need to get some extra support into the Win32 engine,
such that it can handle both GDI and PMGRE requests:
1. callbacks for SetupDC (this is a handshake that takes place when
some of the graphic objects are to be repainted after a window change).
This callback needs to (a) know the windows is a PM-style and (b) know
how to call it.
2. GDI interface for unicode. This is not speced yet, but bobmu
says it is in the plan for 1st release.
3. Well define how a DC is setup to be a PM-style DC. This will
need to propagate correctly in order for the final pixels to be drawn
correclty. This is needed, to overcome a few differnces between PM
graphics and WIN grahpics:
- line drawing: end-point exclusive vs start-point exclusive.
- coordinate system: upper left vs lower left corner origin.
- rectangle fill algorithm (Hard, too technical for this memo).
- line drawing rounding (ditto).
4. Callbacks for VisRgnCallback need to call PMWIN as well
as USER.
5. Need to make sure that NTDDI supports PM-style drawing, and if
modifications needed, do it soon so external developers of drivers
are awarer. Make sure that MS display drivers implement and test the
support.
5.1.2.5 PMWIN:
-------------
We will look hard into using PPM PMWIN code.
The order of work will be:
o make it work with WIN32 components, as a server only (i.e. clients
will link in as a DLL). This will acheive the integration needed
with the input, GDI, display of WIN32/NT.
o After making it completely work, we will start pulling off
server-side APIs to the client DLL as possible. Given the approach
to GPI/GRE above, it could be straight forward. We will need to
work closely with the WIN32/NT USER and GDI group on the remoting
issue.
o We will also look at the WIN32 performance analysis on going
what optimizations to do to PM32 (e.g. batch GPI calls). We will
adjust or do differently, as fit and as time allows.
Input synergy: PPM desktop sits on top of the main windowing system.
So while WIN32 is active, you see a little icon of PM that you can
switch to. A specific key-stroke or a mouse click on the icon will
open and maximize the PM desktop. The support underneath is provided
by WIN32 USER: a router sits below all desktops and an API is avail
for switching desktops. For the WIN32 USER, PM is another (though somewhat
special) client window. The input module of PM will be modified to
unwrap the windowing messages back into raw input messages, so the
rest of PMWIN is unaware. It is yet unclear to us whether additional
support is required from USER, to get all the low-level info up to
the PM desktop input thread.
PMWIN and input system Integration notes:
----------------------------------------
- we will need the input support described above.
- SPB calls need to be exported, such that PM 'desktop' gets the
information on PM apps' video output while in background.
5.1.2.6 PM environment services:
-------------------------------
Here included all the PM apps that come with the retail OS/2 2.0. Those
will have to be ported and to be integrated with the equivalent WIN32s.
- PM SHELL *
- PM Control Pannel *
- PM spooler (print manager) **
- PM File Manager **
- PM task manager ***
- PMVIO (console handler) **
* must be integrated to some extent, and scrap redundant functionality.
Examples: the shell serves as a process frame for some system thread.
We might want to use the WIN32 server process for this same purpose.
OR - the control pannel. We would like to see there stuff related to
the virtual desktop of PM (e.g. background color), but not stuff related
to the physical desktop (e.g. printer driver setup).
**want to scrap, and leave the user with one utility for the whole system
e.g. one file manager, one print manager - that of WIN32. For the file
manager, there are some questions to be answered, like 'will the WIN32
file manager be able to run PM apps, OS2 apps? how about managing EAs?
***should see only PM tasks. Might be neat to have a quick, double-click way,
to get at the system print manager (this is equivalent to the sequence
'get out of PM desktop; double-click on print manager under WIN32'.
5.1.3 pros and cons for the split screen approach:
---------------------------------------------------
+ avoids the clash between PMWIN design and windows USER design.
+ can be made robust and portable (see papers on WIN32 client-server
discussion).
+ best attempt to use existing portable code.
+ Allow IBM and MS to proceed with shell work without a hopeless
integration effort between IAYF and CUA.
- split screen groups is ugly and can confuse the user.
- duplicate functionality (control pannel, shell etc.) need
to be avoided, otherwise more confusion and bugs.
- no way to communicate between PM apps and WIN apps (a-la DDE).
5.2 Non-portable, split screen groups:
--------------------------------------
Same as 4.1, except we can look at using larger chunks of Cruiser code.
To get to the extreme, we could do a port of Cruiser PM code to work on
NT/X86...
This will not achieve any long term goals (stable, portable code base).
but might achieve a better 16B compatibility. Does not look like a good
match for 'OS2 on NT' goals.
5.3 Portable, joint screen groups:
---------------------------------
HolePort. This is the user's dream. The Porthole experience says you
can do 80% of it, but the rest is a night mare. Now, the case is similar
but some differences are note worthy:
- the base is a portable, WIN32 environment, not the PM environment that
is the base for PortHole.
- there are very few PM32 apps, so less pecularities you need to deal
with.
- if the PM 32 APIs are written strict enough, we might be able to force
PM32 apps to behhave nicely.
5.3.1 Pros and Cons for HolePort:
--------------------------------
+ same screen group, DDE, Clipboard - all those are in the reach.
+ no need for multiple file managers, spoolers, shells... the PM
work boils down to PMWIN->USER layer, on client and server, plus
GPI->GDI.
- risky, given the porthole experience.
- Performance might be slow.
We will not go Holeport, but rather work on both USER and PMWIN to make
them work together. The approach will be that every PM windows will have
it's frame be a WIN32/USER windows, while inner (clipped to it) windows
will not be known to USER, and managed solely by PMWIN. (see separate
note on this).
6 Synergy with WIN32/NT:
-----------------------
Our intent is to find a way to work with the WIN32/NT team, such that
we can work mostly indpendantly, without using a lot of the WIN32/NT
time, and also to use as much as possible the development environment
and the code itself.
6.1 Development Environment:
----------------------------
We will pick up the OS/2 base work (os2ss), the PMWIN work and PM engine
work, from where they were back in August 90.
We will enlist in both in the main NT shares (popcorn\razzle and ntos2\ntrel), and get the PMWIN stuff
from \\popcorn\oldraz, the PMGPI stuff from \\oatbran\pm, and the PMGRE stuff
from \\rastaman\pm share incrorporated into a PM subtree of \nt\private SLM
tree. Need to talk with kengr on doing this.
In the first phase we will build the PM server together with the WIN server,
but not in the same place, possibly using the WIN server code as a DLL
(our server will also be a DLL). The idea here is not to put the burden of
building PM every time you build WIN.
6.2 Testing environment:
------------------------
We need to pick up the Cruiser test suites, those already ported to NT
and those that were not ported yet. Need to sync with moshed.
6.3 Technical writing:
---------------------
TBD (both requirements and plan of attack).
7. Task Break Down:
------------------
7.1 Phase 1 - 32 base:
----------------------
Design and integration - this phase is farely indepenent and can
proceed any time. The main design point to be figured out with
the WIN/NT team is the CSR integration:
We need to design how an OS2 application can be
loaded from CMD.exe and WIN task manager. Ideally we will be
able to eliminate CMD32.exe all together at this phase. A WIN32
app should be able to load an OS2 executable and vice versa.
If we need to load LE binaries, then we will implement a loader,
using Cruiser loader source code. This loader will be a PE application
itself, and will be loaded as a shadow when an LE application
is to be loaded. This is an optional design point to figure out
with the NT loader/process people.
For testing we will use the Cruiser test suite.
Task #eng #week Comments
---------------------------------------------------
CSR integration 1 2 (do we need UMTYPE OS2?)
Fix Cruiser CRT 1 2 (find out who own CRT)
Mov IO to console 1 1
Loader for LE 1 16 May not be needed.
Cruiser Named Pipes 1 3
NLS 1 2 (see apistat)
DBCS 1 2 (see apistat)
IOCTLS 1 3 (see ioctls.txt)
IFS apis 1 3 (?) (need to look at steve's design)
Finish MuxWaitSem 1 2 (?) (need to look at)
Finish Mem Apis 1 2 (?) (need to look at)
Session APIs 1 3 (?)
Integration+Test 1 40
Total 2-3 80 (app 2 MY)
Serialization notes:
the CSR integration should be done before the session APIs and the loader.
The APIs items can be done in parallel. NLS/DBCS is a block,
and IFS/IOCTLs too. Component testing can be done any time.
7.2 Phase 2 - 16 VIO+base:
-------------------------
Design, dependancies, integration:
This phase needs phase 1 to be largely complete, and at least the non-APIs
items must be complete. It also depends on NT kernel to provide LDT mgmt
APIs and 16 bit context switch. If an LE loader is not provided by the
previous phase, then the NE loader becomes a major job.
Task #eng #week Comments
---------------------------------------------------
16->32 thunks 1 5
NE Loader 1 8 (given we did the LE loader in phase 1)
16B context switch 1 1 Mostly integration with NT.
LDT management 1 3
16B mem mgmt 1 10
VIO APIs 1 30 (?)
MOU 1 10 (?)
KBD 1 10 (?)
MON 1 5 (?)
16B semaphores 1 4
Signal APIs 1 4
Test+Integration 2 90
Total 3-4 180 (~5MY)
Serialization notes:
The first four items must be completed before any 16 bit program
can be loaded.Thunks should be farely easy and can start way ahead
the rest of the phase. We will take the WIN/DOS thunk compiler.
The API items are mostly independent of each other, although MM and
SEM should be done first, to get simple CRT apps running.
Grouping:
Loader work is an independant task. It makes sense to have the same
group (size 1) do the LDT mgmt and 16 bit context switch.
The VIO/KBD/MOU/MON work should be done by a group that will expertize
in it (upto 2 developers).
Signals, Sem and MM could be done be another group (size 1).
7.3 Phase 3 - PM, split screen groups:
-------------------------------------
Design, dependencies, integration:
PMGRE:
------
Implement, from scratch, a layer that provides OS2 2.0 PMGRE interface
(defined by pmsrc\pmgre\devtable.cxx), that sits on top of WIN32 GDI.
PMGPI:
------
transform the PMGPI IBM code of Cruiser into the NT environment, flatten
it and make it portable.
PMWIN:
------
write a low-level input processing thread that will process WIN32 WM
messages coming into the PM virtual desktop, and feed them into a PM
system queue, to mimiq the native PM environment.
Impose exception handling to make PMWIN robust enough to sit in the
WIN32 server process.
Then take a breath, and integrate the whole chunk of PPM PMWIN code
to run with the new GRE, GPI, input, GDI, DDI - all under one address
space (i.e. client test app linked in).
PM Utilities:
-------------
- PMPRINT - don't do.
- PMCPL - remove stuff that conflicts with the real desktop, port
and test the stuff relevant to a virtual desktop.
- PMSHELL - port. cleanup redundant stuff.
- OS2SM - port and cleanup (need to look at what was done for PPM).
- PMFILE - don't do.
- PMEXEC - port.
Task #eng #week Comments
---------------------------------------------------
Write PMGRE/GDI
port GPI
Complete PMWIN
Hook input USER Recongize the 'PM session', SPB.
GDI2PM callback PM-flavor DC, VisRgn.
16->32 thunks
Client caching
NLS/DBCS
Configuration (os2sys.ini, os2.ini).
PMSHELL
PMCPL
PMEXEC
Test+Integration
Total
Serialization notes:
Grouping:
7.4 Phase 4 - PM, joint screen groups:
-------------------------------------
TBS
8 Open Issues:
--------------
* need to construct a table of all 'platforms' and the relative
compatibility to be achieved on them.
* spooler APIs: do we need to support?
* the PPM layer is implemented on top of NT (not on top of OS/2 base
APIs). Will there be a mismatch when apps are going to use both PM
and OS2 base APIs?
* Can we setup the WIN32 file manager to handle both environments?
(Run, EAs).
