patch 8.2.0149: maintaining a Vim9 branch separately is more workv8.2.0149

Problem:    Maintaining a Vim9 branch separately is more work.
Solution:   Merge the Vim9 script changes.

1 files changed, 344 insertions, 0 deletions

diff --git a/README_VIM9.md b/README_VIM9.md
new file mode 100644
index 000000000..b77d013c0
--- /dev/null
+++ b/README_VIM9.md
@@ -0,0 +1,344 @@
+![Vim Logo](https://github.com/vim/vim/blob/master/runtime/vimlogo.gif)
+
+# What is Vim9?
+
+This is an experimental side of [Vim](https://github.com/vim/vim).
+It explores ways of making Vim script faster and better.
+
+WARNING: The Vim9 script features are in the early stages of development,
+anything can break!
+
+# Why Vim9?
+
+## 1. FASTER VIM SCRIPT
+
+The third item on the poll results of 2018, after popup windows and text
+properties, is faster Vim script.  So how do we do that?
+
+I have been throwing some ideas around, and soon came to the conclusion
+that the current way functions are called and executed, with
+dictionaries for the arguments and local variables, is never going to be
+very fast.  We're lucky if we can make it twice as fast.  The overhead
+of a function call and executing every line is just too high.
+
+So what then?  We can only make something fast by having a new way of
+defining a function, with similar but different properties of the old
+way:
+* Arguments are only available by name, not through the a: dictionary or
+  the a:000 list.
+* Local variables are not available in an l: dictionary.
+* A few more things that slow us down, such as exception handling details.
+
+I Implemented a "proof of concept" and measured the time to run a simple
+for loop with an addition (Justin used this example in his presentation,
+full code is below):
+
+``` vim
+  let sum = 0
+  for i in range(1, 2999999)
+    let sum += i
+  endfor
+```
+
+| how     | time in sec |
+| --------| -------- |
+| Vim old | 5.018541 |
+| Python  | 0.369598 |
+| Lua     | 0.078817 |
+| Vim new | 0.073595 |
+
+That looks very promising!  It's just one example, but it shows how much
+we can gain, and also that Vim script can be faster than builtin
+interfaces.
+
+In practice the script would not do something useless as counting but change
+the text.  For example, re-indent all the lines:
+
+``` vim
+  let totallen = 0
+  for i in range(1, 100000)
+    call setline(i, '    ' .. getline(i))
+    let totallen += len(getline(i))
+  endfor
+```
+
+| how     | time in sec |
+| --------| -------- |
+| Vim old | 0.853752 |
+| Python  | 0.304584 |
+| Lua     | 0.286573 |
+| Vim new | 0.190276 |
+
+The differences are smaller, but Vim 9 script is clearly the fastest.
+
+How does Vim9 script work?  The function is first compiled into a sequence of
+instructions.  Each instruction has one or two parameters and a stack is
+used to store intermediate results.  Local variables are also on the
+stack, space is reserved during compilation.  This is a fairly normal
+way of compilation into an intermediate format, specialized for Vim,
+e.g. each stack item is a typeval_T.  And one of the instructions is
+"execute Ex command", for commands that are not compiled.
+
+
+## 2. PHASING OUT INTERFACES
+
+Attempts have been made to implement functionality with built-in script
+languages such as Python, Perl, Lua, Tcl and Ruby.  This never gained much
+foothold, for various reasons.
+
+Instead of using script language support in Vim:
+* Encourage implementing external tools in any language and communicate
+  with them.  The job and channel support already makes this possible.
+  Really any language can be used, also Java and Go, which are not
+  available built-in.
+* Phase out the built-in language interfaces, make maintenance a bit easier
+  and executables easier to build.  They will be kept for backwards
+  compatibility, no new features.
+* Improve the Vim script language, it is used to communicate with the external
+  tool and implements the Vim side of the interface.  Also, it can be used when
+  an external tool is undesired.
+
+All together this creates a clear situation: Vim with the +eval feature
+will be sufficient for most plugins, while some plugins require
+installing a tool that can be written in any language.  No confusion
+about having Vim but the plugin not working because some specific
+language is missing.  This is a good long term goal.
+
+Rationale: Why is it better to run a tool separately from Vim than using a
+built-in interface and interpreter?  Take for example something that is
+written in Python:
+* The built-in interface uses the embedded python interpreter.  This is less
+  well maintained than the python command.  Building Vim with it requires
+  installing developer packages.  If loaded dynamically there can be a version
+  mismatch.
+* When running the tool externally the standard python command can be used,
+  which is quite often available by default or can be easily installed.
+* The built-in interface has an API that is unique for Vim with Python. This is
+  an extra API to learn.
+* A .py file can be compiled into a .pyc file and execute much faster.
+* Inside Vim multi-threading can cause problems, since the Vim core is single
+  threaded.  In an external tool there are no such problems.
+* The Vim part is written in .vim files, the Python part is in .py files, this
+  is nicely separated.
+* Disadvantage: An interface needs to be made between Vim and Python.
+  JSON is available for this, and it's fairly easy to use.  But it still
+  requires implementing asynchronous communication.
+
+
+## 3. BETTER VIM SCRIPT
+
+To make Vim faster a new way of defining a function needs to be added.
+While we are doing that, since the lines in this function won't be fully
+backwards compatible anyway, we can also make Vim script easier to use.
+In other words: "less weird".  Making it work more like modern
+programming languages will help.  No surprises.
+
+A good example is how in a function the arguments are prefixed with
+"a:". No other language I know does that, so let's drop it.
+
+Taking this one step further is also dropping "s:" for script-local variables;
+everything at the script level is script-local by default.  Since this is not
+backwards compatible it requires a new script style: Vim9 script!
+
+It should be possible to convert code from other languages to Vim
+script.  We can add functionality to make this easier.  This still needs
+to be discussed, but we can consider adding type checking and a simple
+form of classes.  If you look at JavaScript for example, it has gone
+through these stages over time, adding real class support and now
+TypeScript adds type checking.  But we'll have to see how much of that
+we actually want to include in Vim script.  Ideally a conversion tool
+can take Python, JavaScript or TypeScript code and convert it to Vim
+script, with only some things that cannot be converted.
+
+Vim script won't work the same as any specific language, but we can use
+mechanisms that are commonly known, ideally with the same syntax.  One
+thing I have been thinking of is assignments without ":let".  I often
+make that mistake (after writing JavaScript especially).  I think it is
+possible, if we make local variables shadow commands.  That should be OK,
+if you shadow a command you want to use, just rename the variable.
+Using "let" and "const" to declare a variable, like in JavaScript and
+TypeScript, can work:
+
+
+``` vim
+def MyFunction(arg: number): number
+   let local = 1
+   let todo = arg
+   const ADD = 88
+   while todo > 0
+      local += ADD
+      --todo
+   endwhile
+   return local
+enddef
+```
+
+The similarity with JavaScript/TypeScript can also be used for dependencies
+between files.  Vim currently uses the `:source` command, which has several
+disadvantages:
+*   In the sourced script, is not clear what it provides.  By default all
+    functions are global and can be used elsewhere.
+*   In a script that sources other scripts, it is not clear what function comes
+    from what sourced script.  Finding the implementation is a hassle.
+*   Prevention of loading the whole script twice must be manually implemented.
+
+We can use the `:import` and `:export` commands from the JavaScript standard to
+make this much better.  For example, in script "myfunction.vim" define a
+function and export it:
+
+``` vim
+vim9script  " Vim9 script syntax used here
+
+let local = 'local variable is not exported, script-local'
+
+export def MyFunction()  " exported function
+...
+
+def LocalFunction() " not exported, script-local
+...
+```
+
+And in another script import the function:
+
+``` vim
+vim9script  " Vim9 script syntax used here
+
+import MyFunction from 'myfunction.vim'
+```
+
+This looks like JavaScript/TypeScript, thus many users will understand the
+syntax.
+
+These are ideas, this will take time to design, discuss and implement.
+Eventually this will lead to Vim 9!
+
+
+## Code for sum time measurements
+
+Vim was build with -O2.
+
+``` vim
+func VimOld()
+  let sum = 0
+  for i in range(1, 2999999)
+    let sum += i
+  endfor
+  return sum
+endfunc
+
+func Python()
+  py3 << END
+sum = 0
+for i in range(1, 3000000):
+  sum += i
+END
+  return py3eval('sum')
+endfunc
+
+func Lua()
+  lua << END
+    sum = 0
+    for i = 1, 2999999 do
+      sum = sum + i
+    end
+END
+  return luaeval('sum')
+endfunc
+
+def VimNew()
+  let sum = 0
+  for i in range(1, 2999999)
+    let sum += i
+  endfor
+  return sum
+enddef
+
+let start = reltime()
+echo VimOld()
+echo 'Vim old: ' .. reltimestr(reltime(start))
+
+let start = reltime()
+echo Python()
+echo 'Python: ' .. reltimestr(reltime(start))
+
+let start = reltime()
+echo Lua()
+echo 'Lua: ' .. reltimestr(reltime(start))
+
+let start = reltime()
+echo VimNew()
+echo 'Vim new: ' .. reltimestr(reltime(start))
+```
+
+## Code for indent time measurements
+
+``` vim
+def VimNew(): number
+  let totallen = 0
+  for i in range(1, 100000)
+    setline(i, '    ' .. getline(i))
+    totallen += len(getline(i))
+  endfor
+  return totallen
+enddef
+
+func VimOld()
+  let totallen = 0
+  for i in range(1, 100000)
+    call setline(i, '    ' .. getline(i))
+    let totallen += len(getline(i))
+  endfor
+  return totallen
+endfunc
+
+func Lua()
+  lua << END
+    b = vim.buffer()
+    totallen = 0
+    for i = 1, 100000 do
+      b[i] = "    " .. b[i]
+      totallen = totallen + string.len(b[i])
+    end
+END
+  return luaeval('totallen')
+endfunc
+
+func Python()
+  py3 << END
+cb = vim.current.buffer
+totallen = 0
+for i in range(0, 100000):
+  cb[i] = '    ' + cb[i]
+  totallen += len(cb[i])
+END
+  return py3eval('totallen')
+endfunc
+
+new
+call setline(1, range(100000))
+let start = reltime()
+echo VimOld()
+echo 'Vim old: ' .. reltimestr(reltime(start))
+bwipe!
+
+new
+call setline(1, range(100000))
+let start = reltime()
+echo Python()
+echo 'Python: ' .. reltimestr(reltime(start))
+bwipe!
+ 
+new
+call setline(1, range(100000))
+let start = reltime()
+echo Lua()
+echo 'Lua: ' .. reltimestr(reltime(start))
+bwipe!
+
+new
+call setline(1, range(100000))
+let start = reltime()
+echo VimNew()
+echo 'Vim new: ' .. reltimestr(reltime(start))
+bwipe!
+```