Transfering From Vocaloid 4 - Looking for Resources

brogar2000 · 2023 年 12 月 27 日午前 3:28

I think I’ve posted here before, but I can’t remember. In case I haven’t mentioned it already, I am visually impaired - I can only see lights and shadows. Until recently, Vocaloid (versions 1-4) was the only vocal synth program I was able to use because of its keyboard shortcuts and job plugins. Until last week, nobody knew if Synth V was accessible for us. The light version isn’t, and that alone was justification enough not to buy the pro version.

Recently, I’ve been wondering if it’s possible to make Synth V accessible by scripting it. On a whim at about 3:00 AM, I decided to buy it and explore it a little bit. As it turns out, the pro version is accessible for us, and is in fact much more accessible than Vocaloid… To a certain extent. There are a few features that Synth V lacks which prevents me from switching away from the Vocaloid editor entirely, and I’ll cover those below. If anyone can point me to some resources, I would greatly appreciate it.

Before I continue, I should mention that Synth V’s interface is very interesting when using a screen reader. I had to have a sighted person assist me in adding keyboard shortcuts and changing settings. It is entirely possible that we missed a few settings. If I missed something, please forgive me.

Inserting Notes

In Vocaloid, I set most of the keyboard shortcuts myself. I had Numpad Plus and Numpad Minus set to move forward and backward by 1 measure. Numpad 4 and Numpad 6 moved backwards and forwards by grid division respectively. I would move to the spot I wanted to insert a note at, then I would press I to insert it at that spot. Highlighting the note with the arrow keys, I would then use Ctrl + Numpad 4/6 to adjust the length of the note. Numpad 4/6 controlled the right edge of the note, while Numpad 1/3 controlled the left edge. If I made the note too long and it overlapped with other notes, I could just go into the Job menu and fix the overlapped notes.

I downloaded Clare’s scripts for Synth V, and using those, I was able to add keyboard shortcuts for selecting notes with the arrow keys. I was also able to set Numpad 8/2 to move notes up/down 1 semitone respectively. Unfortunately, I haven’t found a way to insert notes at the current grid division using a keyboard shortcut, nor have I found a way to increase note length using the keyboard. This means I have to use Vocaloid to make the song, then convert it to a Synth V project. If I mess something up, I have to go back into Vocaloid, fix the issue, then re-export the project.

Adjusting Parameter Curves

I use a Vocaloid job plugin called Draw Control, because I can’t draw the parameter lines. It gives me these options:

Control Type: a combo-box listing the various parameters that I can pick from
Begin: a text field to type the initial value of the parameter, (usually from 0 to 127)
End: another text field to type the value it will be set to at the end of the selected part of the timeline, which is usually based on selected notes for me
Curve Type: a combo-box which allows you to select between Linear, Log, and Exp
- I don’t really know what this does, so I leave it at the default, which is Linear
Return to Default Value: a check-box which determines if the value will return to the default at the end of your selection, or if it will stay at that value for the duration of the current part
OK and Cancel Buttons

Alternatively, I sometimes opt to go for a job plugin called Adjust Controls. This plugin gives me check-boxes for each parameter, which I can use to adjust multiple parameters at the same time.

I am aware that a user who goes by the name Dcuny made a script for Synth V called Expression Curves. I tried to use the script myself, and unfortunately, it wouldn’t work for me. You see, I would still have to adjust a visual control point by dragging it, which I am unable to do since I can’t use the mouse. I’m basically looking for the Synth V equivalent of the job plugin(s) I’m using with Vocaloid.

Vibrato

Vocaloid’s default vibrato function is very limited. Instead, I highlight a note with the arrow keys, then use a job plugin called Sign Vibrato. It gives me 2 text fields - one to type in the depth value, and one to type in the rate. It’s as simple as that. It doesn’t really work well if you try to select multiple notes at once, but for single notes it’s great. If I have issues with multiple notes, I just combine them all with the Join Notes job plugin, apply the vibrato, then split the notes back up again with the Divide Notes Equally plugin if need be.

I have no idea how to use the vibrato function in Synth V. Although I can move through the interface buttons with my screen reader, it does me no good, because they have no labels for the screen reader to identify. They all say graphical button, or they say nothing at all. I don’t know where to even begin to explain how to fix this particular problem. Perhaps a script can be made that can change vibrato depth and rate in a similar way to the Sign Vibrato job plugin?

What am I Able to do?

Thanks to Clare’s scripts, a lot more than I initially thought.

I can set note-independent lyrics and phonemes
I can move the play head forward and back by what I’m assuming is a measure, but I’m not sure because the play head doesn’t return to last position
I can preview notes, which is paramount for knowing exactly where I am and what I’m doing in a project
I can move through tracks - something which you can apparently do in Vocaloid with keyboard shortcuts, though I haven’t been able to get those working
I can probably do other things I’m not thinking of off the top of my head.

Without Clare’s scripts, I would never have gotten as far with Synth V as I did. Besides those, I am able to divide notes using the default script that comes with the program. Sadly, until I find a way to select multiple notes with the keyboard, I will not be able to use the join notes script.

Conclusion

I hope everything I said made sense. I know this was a lot of information. Like I said before, there are things I’m sure I’m missing. I’m very much new to Synth V, and the only sighted person available to help me knows nothing about vocal synths at all. Thank you in advance for any help you can provide.

claire · 2023 年 12 月 28 日午後 3:22

I’ll try to answer the questions I can. I think these are the main limitations though:

The UI controls do not have proper labels for a screenreader
The slider controls cannot be modified without a mouse
Dialog boxes opened by scripts cannot be interacted with until clicked (so prompting for user input is difficult)
Some features are still not accessible via scripting, for example, changing the grid snap setting

So this means that I think any scripts would need to make incremental changes rather than prompting for a value? At least until Dreamtonics can address the issue, or unless you’ve found a way to navigate the dialog boxes that I haven’t.

Adjusting Parameter Curves

Regarding parameter curves, a script similar to the Draw Control plugin seems like it would be possible. Whether it would be usable would mainly depend on whether the dialog controls are screenreader-friendly.

Scripts can read the clipboard, so that’s potentially a workaround, but might be very awkward to use since you’d need to tab over to a text editor repeatedly, copy the inputs in some sort of consistent format, and then run the script… pretty bad usability, but maybe there’s something there…

Vibrato

SynthV Studio has a few different “pitch modes”. They’re under the Note Properties panel, but I’m assuming they’re read as some of those nondescript “graphical buttons” you mentioned. The mode can be modified with a script though, so I’ll add that to my to-do list.

“Sing” mode will automatically add pitch deviations and vibrato to the notes based on the machine learning analysis of the original vocalist. “Rap” mode is the same, but for rapping instead of melodic singing. Both these modes are considered “auto pitch” modes, meaning that when a note is changed they will recalculate the pitch in an attempt to fit the new context (but this can also mean they overwrite your manual changes).

“Manual” mode is basically what Vocaloid offers, with pitches “snapped to” the notes and some minor transition curves and basic sine-wave vibrato; basically a blank slate for customization. This is the one that will probably be easiest to work with using a screenreader, because the vibrato is controlled by sliders that perhaps could be tweaked incrementally with scripts. Of course, ideally the values would be readable and settable using a screenreader, which would free up a lot of hotkeys required by the scripting approach.

One thing to note is that when enabling Manual mode, the Sing/Rap mode pitch deviations will stick around as parameters, so you might need to reset the pitch for the selected notes to get rid of those (there’s a default option for this in the Modify menu which can be hotkeyed). The default behavior when changing pitch modes can be changed in the Settings panel, with a dropdown menu “Note Default Pitch Mode” and two checkboxes for whether to keep the pitch curves when moving from Sing to Manual mode or vice versa.

Conclusion

I’m really glad that my scripts were able to help, and I’ll definitely be coming up with a list of additions to tackle some of these other obstacles. I can’t make any promises of timeframe, but I’ll be sure to post here once I’ve made some progress.

In the meantime, I’ll be sending another email to Dreamtonics about some of these limitations. I’ve raised a few of them before, but I wasn’t aware of some of the more egregious ones (like the screenreader incompatibility of many of the basic controls). I’ll refer them back to this topic for review, but if you’d like you can also send an email to [email protected].

brogar2000 · 2023 年 12 月 30 日午前 1:58

Thanks for the response. I have additional details I can provide, so I will respond to some of your points.

The UI controls do not have proper labels for a screenreader

Correct. Most UI controls do not have proper labels. However, there are some that do have proper labels, or are obvious enough to use without them. For example, while the voice combo-box is not labeled, it’s purpose is somewhat obvious, because it’s default item is labeled “Not Set,” and when clicked, it shows the available voices. A lot of the options in the settings panel are accessible and have labels most of the time as well, sadly with the exception of the keyboard shortcuts interface, which I had to have help from my brother to set up.

The slider controls cannot be modified without a mouse

Also sadly correct. It’s also worth mentioning that my screen reader (NVDA), has something called object navigation, which I can use to navigate through buttons that can’t necessarily be focused on properly. Anyway, object navigation skips right over most of the sliders, as if they never existed. I can tab between them, but they all say unknown. There is another screen reader called Jaws that I have tested with Synth V, and Jaws’s object navigation equivalent is able to move to the sliders, though it only says custom when you move across them. I haven’t tested it with Microsoft Narrator, but I imagine it’s much the same.
I have also been wanting to test Synth V on Mac OS with Voice Over and/or Linux with the Orca Screen-Reader, but the TOS says that I can only have Synth V installed on one computer. I’m a bit wary about breaking it for fear of losing my license, which I have paid a considerable amount of money for. I’d also rather not pay for a second license just to run the same program on a second computer, to be honest. Perhaps I can work something out with Dream Tonics?

Dialog boxes opened by scripts cannot be interacted with until clicked

This is true, although object navigation can interact with these dialogues, so it’s only a miner annoyance.

Adjusting Parameter Curves

Regarding parameter curves, a script similar to the Draw Control plugin seems like it would be possible. Whether it would be usable would mainly depend on whether the dialog controls are screenreader-friendly.

If it uses the same type of combo-boxes, text fields, and check-boxes that the main interface does, they would be usable. If it uses any of Synth V’s custom slider controls, however, it would not be natively accessible at all.

Scripts can read the clipboard, so that’s potentially a workaround, but might be very awkward to use since you’d need to tab over to a text editor repeatedly, copy the inputs in some sort of consistent format, and then run the script… pretty bad usability, but maybe there’s something there…

That is a potential option to consider, but you’re right in that it would be a bit cumbersome to use. I don’t think that’ll be necessary though.

Vibrato

“Manual” mode is basically what Vocaloid offers, with pitches “snapped to” the notes and some minor transition curves and basic sine-wave vibrato; basically a blank slate for customization. This is the one that will probably be easiest to work with using a screenreader, because the vibrato is controlled by sliders that perhaps could be tweaked incrementally with scripts. Of course, ideally the values would be readable and settable using a screenreader, which would free up a lot of hotkeys required by the scripting approach.

I rarely, if ever, opt to use the auto-singing feature, as I like to tune my projects manually most of the time. It would be fantastic to use the rap feature though, because I’m not a good rapper at all.

Conclusion and Additional Context

In the meantime, I’ll be sending another email to Dreamtonics about some of these limitations. I’ve raised a few of them before, but I wasn’t aware of some of the more egregious ones (like the screenreader incompatibility of many of the basic controls). I’ll refer them back to this topic for review, but if you’d like you can also send an email to [email protected].

Unfortunately, when I emailed them a couple years ago about this issue, I did not receive a response back from them. Although, this could be because I didn’t have any idea on what needed to be done to fix some of these issues, and they likewise had no idea what to do. Around that time, I had no idea how vocal synths worked. Now I have a better idea of what issues exist with the program, and so it is easier to compile resources on what exactly needs to be done to fix said issues. Speaking of which, I have some final details I’d like to cover:

Would it be possible to make a script that, when you select a note, it plays a beep sound at the same key and duration as the selected note? Currently, you have to preview notes every time you move to them with a hotkey, and because we’re dealing with AI voices, the response time is a little slow. You can make it play a beep sound if the voice is not set or unloaded, but this means you have to unset and reset the voice every time. The preview notes feature also plays a little bit of the notes surrounding the current one you just previewed. Personally, I think it would be useful to have both.
Can a script be made that sets focus to the menu bar when the alt key is pressed, so that I don’t have to use object nav to click on the menu bar?
My friends and I were discussing potential ways to make the interface more accessible. NVDA, like Synth V, has a scripting feature. It can be used to add more features to the screen reader, or make applications accessible that weren’t before. We thought that, if the Lua VM that Synth V uses is not sand-boxed, we could make both an NVDA addon and a Synth V script that can communicate with each other to relay important information about the interface back to the user. If this is possible, the entire Synth V interface could be made accessible. There is an alternative method that we could use where we can have the screen reader screenshot the window and relay information from the screenshot, which has definitely been done before, but that is a rather hacky method and is absolutely not preferred.
I submitted an issue about the note insert hotkey. If you like, I can start submitting issues about all these other potential scripts so that you can track them easier. BTW, for anyone reading this topic who doesn’t know about Clare’s scripts, you can find them (here)[GitHub - claire-west/svstudio-scripts: A collection of scripts for Synthesizer V Studio Pro.].

I will be referring my friends to this thread so they can review it and give additional context. They are programmers, so they might be able to fill you in on something that I’m not as familiar with. However, I intend on studying programming, as this is a potential career path I’m going to look at after college. Perhaps we can all work together to solve this.

Thank you for all the help and information you’ve provided so far. If you wish, I might be able to throw together a quick video sometime to demonstrate what works and what doesn’t work with my screen reader, if you think that would help provide some more context on your end.

claire · 2024 年 1 月 1 日午後 9:54

Okay, it’s definitely helpful to know that the sliders and fancy controls are the main issue, and that dialogs with text boxes might be usable. This means that some slider-heavy settings can be exposed by a script using a dialog with text boxes, where it pre-populates the fields with the current value and allows changing them by simply typing in the value.

As for the TOS, yeah, it’s a bit of an odd situation. The Dreamtonics voice databases all allow three concurrent devices, while the Pro edition says only one. The product activation functionally follows the same pattern as is used by Vocaloid (ie. each code has three uses, which are returned upon deactivation, and the software only checks with the servers at the time of activating or deactivating).

I think the internal grid setting idea is feasible, it’s just a matter of making the scripts in a way that they can refer to the value, as well as offering a way to increase/decrease/check the setting. This means the scripts likely need to save and refer to a shared configuration file, meaning they need to be written in Lua (since JavaScript can’t access files). This isn’t a huge obstacle, it just means existing JS scripts will need to be rewritten in Lua to make use of this.

Regarding the beep idea, I’m not sure there’s any way to do this. I spent some time experimenting with loading a C or C++ library as a Lua module, but I wasn’t able to find a way to do it reliably. If I can find a way to do that reliably then it would make a lot of things possible, but as-is the small C module I wrote seems to work sometimes, while outright crashing the software other times, with no clear reason or cause.

The script bindings for SynthV Studio are very focused on the in-editor objects; notes, tracks, groups, and parameters, so they don’t offer direct access to some of the higher level things like the top menus.

While I was tinkering with the C library idea, I also did a quick proof-of-concept with triggering text-to-speech from within a script. My thought was that scripts could provide some level of auditory feedback when executed (like informing the user of the new grid snap setting when it is increased/decreased). These were my findings:

On Windows, calling any command line tool forces SynthV Studio into the background. Sadly, this is a side-effect of the windows command line itself, so there’s no trivial fix from within Lua.
That said, I did find a workaround which involves saving a file with the text to be read out. An external program could watch the file and read out the text any time the file contents change.
- If possible an NVDA script/addon could watch this file and read out any changes
- An alternative would be a simple standalone app which would run in the background and invoke some form of text-to-speech for any change. The simplest option I can think of would be an Electron app, which is a bit overkill for the use case, but could leverage the browser-based Web Speech API.
I asked an acquaintance to check the functionality on a Mac, since I don’t have one, and they said it worked seamlessly, without de-focusing SynthV Studio. The test used the MacOS say command, and since MacOS is UNIX-based, I suspect the Linux spd-say command might behave similarly. This means we can likely invoke text-to-speech directly from within a Lua script.

So, some degree of running commands on MacOS or Linux is possible in a seamless manner, but Windows is much more clunky. The file-watching approach definitely allows for relaying of information between scripts and some other software.

This file-watching method of relaying information might also enable the “beep” idea, since writing a duration and pitch to a file for some external tool to play would be straightforward.

As for the GitHub issues, it might be simpler to have a single topic with a list of things to implement, but either way works.

For now I’ll start with the initial request of an “insert note” script and ones to make the selected note longer/shorter.

brogar2000 · 2024 年 1 月 3 日午前 11:26

So my friend said he would like to come in and participate in the thread, but he wants to look at a few Synth V scripts first so he can understand how the syntax works. He basically wants to make sure he knows what he’s doing first. He was wondering if there was a Lua console so he could test things out, but I told him that I don’t think there is. Everything he knows about the scripting engine comes from my own limited knowledge of it, so I pointed him towards your scripts for reference.

He had a message for you regarding your C issue, which I will quote below:

Lua can’t load any old C code, it can only load C code designed to be loaded by Lua, that’s why you can’t get that working. Your DLL has to have a specific function with a specific signature in it, that kicks off the whole process.

Update: That message was addressed to me, because he thought I was the one trying to write C code. Still, hopefully that was helpful anyway.

He also sent me (this page about testing accessibility for Windows apps)[Accessibility testing - Windows apps | Microsoft Learn], and that page includes a link to the download page for Microsoft’s Accessibility Insights tool. I downloaded it and tested it myself, and it’s pretty cool. It allows you to scan the entire window or specific controls within that window to determine their accessibility status. This is perfect for sighted people, because it allows you to see if a control is screen-reader friendly without having to use a screen-reader yourself.

Not surprisingly in Synth V’s case, when I performed a scan of the window, there were a lot of custom controls that the Accessibility Insights tool couldn’t recognize. There were some labeled controls of course, and if you want to try this tool you’ll be able to see that, which will help you understand what we’re dealing with more effectively. It might also help you understand whether your scripts work with screen-readers or not, and you’ll be able to make adjustments accordingly.

BTW, I tested this tool with Vocaloid 4 as well. I learned that most of it’s controls don’t even show up in the Insights tool, which means they’re painted on the screen, which technically means they aren’t real controls. This in tern means that Synth V is indeed more accessible than Vocaloid to a certain extent, which is fantastic on our part. We most likely won’t have to do any fancy OCR stuff to get it to work. Of course, we will still need to script things both through Synth V and through NVDA, but that’s not such a big deal.

In terms of dialogues not being focusable, an NVDA add-on will be able to fix that fairly easily. I’m told add-ons allow you to modify items in the accessibility tree, which means you can label things that weren’t labeled before, among other things. We will also be able to fix the menu bar issue through the add-on as well. I’m going to be studying Python over the next few months so I’ll be able to start working on this add-on myself.

All things considered, I’m fairly optimistic about this little accessibility project. Once again, thank you for the information you’ve been able to provide so far; it’s been very helpful.

claire · 2024 年 1 月 4 日午前 12:09

I’ll share my C code here in case someone more familiar with C and Lua can get it working more reliably, but it could also be that Dreamtonics didn’t expect people to try this and therefore didn’t set up the environment in a way that facilitates it.

To keep things simple (and follow the few examples I could find online) I only started by exposing a single function that adds two numbers.

With my test script just trying to import the module and add two numbers together, I had some cases where it worked fine, and others where it crashed the host software, unfortunately with no obvious difference between the two scenarios.

#include <lua.h>
#include <lauxlib.h>

static int sum(lua_State* L) {
    lua_Number a = luaL_checknumber(L, 1);
    lua_Number b = luaL_checknumber(L, 2);
    lua_pushnumber(L, a + b);
    return 1;
}

static luaL_Reg const mylib[] = {
    {"sum", sum},
    {NULL, NULL}
};

__declspec(dllexport) int luaopen_synthvc(lua_State* L) {
    luaL_newlib(L, mylib);
    return 1;
}

Test Lua for SynthV Studio:

SCRIPT_TITLE = "Test LUA Script with C module"

function getClientInfo()
    return {
        name = SV:T(SCRIPT_TITLE),
        author = 'claire',
        versionNumber = 1,
        minEditorVersion = 65537
    }
end

function main()
  synthvc = require 'synthvc'
  local result = synthvc.sum(1,2)
  SV:showMessageBox(SV:T(SCRIPT_TITLE), result)
  SV:finish()
end

brogar2000 · 2024 年 3 月 6 日午前 2:17

Update: Thanks to Clare, I just learned that parts of the note properties are accessible already. I’ll have to do some more testing, but so far I can confirm the tab names, (i.e., pitch, language, etc.), are accessible. You can also tab to the language combo-box and set the language for the currently selected note. Do keep in mind that the combo-box is labeled language, so you should take care not to confuse it with the main voice language combo-box, which affects the entire group of notes, or however you refer to that. Also, you need to be sure to tab to a slider in the window in order for the keyboard shortcuts to work. Don’t worry; currently the arrow keys can’t adjust sliders. I do think it’s strange that you can’t use the arrow keys properly if you’re positioned on a button, but you can if you’re on a custom control. Oh well. If it works, it works.

Transfering From Vocaloid 4 - Looking for Resources

Inserting Notes

Adjusting Parameter Curves

Vibrato

Other Issues and Questions

What am I Able to do?

Conclusion

Adjusting Parameter Curves

Vibrato

Other Issues and Questions

Conclusion

Adjusting Parameter Curves

Vibrato

Other Issues and Questions

Conclusion and Additional Context